./testrunner -i /tmp/testexec.45135.ini -p gsi_type=plasma,initial_version=6.6.5-10080,upgrade_to=7.1.0-2434,get-cbcollect-info=True,java_sdk_client=True,reset_services=True,get-cbcollect-info=True,bucket_storage=couchstore -t gsi.upgrade_gsi.UpgradeSecondaryIndex.test_online_upgrade,before=create_index,in_between=query:verify_query_result-query_with_explain,after=query:verify_query_result-drop_index,doc_ops=True,update_ops_per=.5,nodes_out=1,nodes_out_dist=index:1,standard_buckets=1,dataset=default,groups=simple,nodes_init=4,services_init=kv:n1ql-kv-index-index,doc-per-day=10,init_nodes=False,gsi_type=plasma,initial_version=6.6.5-10080,upgrade_to=7.1.0-2434,get-cbcollect-info=True,java_sdk_client=True,reset_services=True,bucket_storage=couchstore Test Input params: {'before': 'create_index', 'in_between': 'query:verify_query_result-query_with_explain', 'after': 'query:verify_query_result-drop_index', 'doc_ops': 'True', 'update_ops_per': '.5', 'nodes_out': '1', 'nodes_out_dist': 'index:1', 'standard_buckets': '1', 'dataset': 'default', 'groups': 'simple', 'nodes_init': '4', 'services_init': 'kv:n1ql-kv-index-index', 'doc-per-day': '10', 'init_nodes': 'False', 'gsi_type': 'plasma', 'initial_version': '6.6.5-10080', 'upgrade_to': '7.1.0-2434', 'get-cbcollect-info': 'True', 'java_sdk_client': 'True', 'reset_services': 'True', 'bucket_storage': 'couchstore', 'ini': '/tmp/testexec.45135.ini', 'cluster_name': 'testexec.45135', 'spec': 'collection-upgrade-gsi', 'conf_file': 'gsi/collection-upgrade-gsi.conf', 'num_nodes': 7, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/centos-p0-2i-vset00-00-upgrade-6.6.x/logs/testrunner-22-Mar-08_03-58-01/test_1'} lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) pytests/basetestcase.py:3160: DeprecationWarning: invalid escape sequence \ copy_servers, root_cn='Root\ Authority', type="openssl", lib/couchbase_helper/data_analysis_helper.py:905: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" ./pytests/security/x509main.py:80: DeprecationWarning: invalid escape sequence \. ''' /data/workspace/centos-p0-2i-vset00-00-upgrade-6.6.x/scripts/install.py:1072: DeprecationWarning: invalid escape sequence \d if re.match('\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', server.ip): pytests/security/SecretsMasterBase.py:63: DeprecationWarning: invalid escape sequence \. status, ipAddress = subprocess.getstatusoutput("ifconfig eth0 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | awk '{print $2}'") test_online_upgrade (gsi.upgrade_gsi.UpgradeSecondaryIndex) ... 2022-03-08 03:58:01,858 - root - INFO - ============== EventingBaseTest setup has started ============== 2022-03-08 03:58:01,858 - root - INFO - Starting Test: test_online_upgrade 2022-03-08 03:58:01,860 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:58:01,968 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:58:02,264 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:02,585 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:02,591 - root - INFO - -->is_ns_server_running? 2022-03-08 03:58:02,604 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 03:58:02,604 - root - INFO - ============== basetestcase setup was started for test #1 test_online_upgrade============== 2022-03-08 03:58:02,618 - root - INFO - list of index nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 03:58:02,620 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:58:02,726 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:58:03,010 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:03,326 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:03,334 - root - ERROR - POST http://172.23.123.61:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-03-08 03:58:03,339 - root - INFO - /diag/eval status on 172.23.123.61:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-08 03:58:03,339 - root - INFO - running command.raw on 172.23.123.61: zgrep "panic" API is accessible from localhost only/indexer.log* | wc -l 2022-03-08 03:58:03,417 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/indexer.log*.gz: No such file or directory'] ... 2022-03-08 03:58:03,431 - root - INFO - list of kv nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 03:58:03,433 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:58:03,539 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:58:03,839 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:04,156 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:04,163 - root - ERROR - POST http://172.23.123.61:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2022-03-08 03:58:04,163 - root - INFO - /diag/eval status on 172.23.123.61:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-08 03:58:04,164 - root - INFO - running command.raw on 172.23.123.61: zgrep "panic" API is accessible from localhost only/projector.log* | wc -l 2022-03-08 03:58:04,246 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/projector.log*.gz: No such file or directory'] ... 2022-03-08 03:58:04,247 - root - INFO - Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2022-03-08 03:58:09,345 - root - INFO - initializing cluster 2022-03-08 03:58:09,347 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:58:09,458 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:58:09,737 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:10,049 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:10,059 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:58:10,159 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:58:10,450 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:10,757 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:10,819 - root - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:10,898 - root - INFO - Running systemd command on this server 2022-03-08 03:58:10,898 - root - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service 2022-03-08 03:58:12,407 - root - INFO - command executed successfully with root 2022-03-08 03:58:12,408 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:12,487 - root - INFO - Couchbase stopped 2022-03-08 03:58:12,487 - root - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:58:12,505 - root - INFO - command executed successfully with root 2022-03-08 03:58:12,505 - root - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:58:12,560 - root - INFO - command executed successfully with root 2022-03-08 03:58:12,562 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:58:12,663 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:58:12,938 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:13,246 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:13,271 - root - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:13,271 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:13,309 - root - INFO - Starting couchbase server 2022-03-08 03:58:13,414 - root - INFO - Running systemd command on this server 2022-03-08 03:58:13,415 - root - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service 2022-03-08 03:58:13,438 - root - INFO - command executed successfully with root 2022-03-08 03:58:13,439 - root - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:58:18,444 - root - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:58:18,476 - root - INFO - command executed successfully with root 2022-03-08 03:58:18,476 - root - INFO - Couchbase server status: [' Process: 42318 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:58:18,476 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:18,553 - root - INFO - process beam.smp is running on 172.23.123.61: with pid 42466 2022-03-08 03:58:18,554 - root - INFO - Couchbase started 2022-03-08 03:58:18,556 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 03:58:18,662 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 03:58:18,954 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:19,265 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:19,276 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 03:58:19,375 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 03:58:19,657 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:20,014 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:20,079 - root - INFO - 172.23.122.122 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:20,188 - root - INFO - Running systemd command on this server 2022-03-08 03:58:20,188 - root - INFO - running command.raw on 172.23.122.122: systemctl stop couchbase-server.service 2022-03-08 03:58:21,697 - root - INFO - command executed successfully with root 2022-03-08 03:58:21,698 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:21,872 - root - INFO - Couchbase stopped 2022-03-08 03:58:21,873 - root - INFO - running command.raw on 172.23.122.122: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:58:21,890 - root - INFO - command executed successfully with root 2022-03-08 03:58:21,890 - root - INFO - running command.raw on 172.23.122.122: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:58:21,944 - root - INFO - command executed successfully with root 2022-03-08 03:58:21,947 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 03:58:22,049 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 03:58:22,331 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:22,633 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:22,695 - root - INFO - 172.23.122.122 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:22,695 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:22,771 - root - INFO - Starting couchbase server 2022-03-08 03:58:22,873 - root - INFO - Running systemd command on this server 2022-03-08 03:58:22,873 - root - INFO - running command.raw on 172.23.122.122: systemctl start couchbase-server.service 2022-03-08 03:58:22,897 - root - INFO - command executed successfully with root 2022-03-08 03:58:22,897 - root - INFO - 172.23.122.122:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:58:27,902 - root - INFO - running command.raw on 172.23.122.122: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:58:27,932 - root - INFO - command executed successfully with root 2022-03-08 03:58:27,932 - root - INFO - Couchbase server status: [' Process: 46840 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:58:27,932 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:28,003 - root - INFO - process beam.smp is running on 172.23.122.122: with pid 46990 2022-03-08 03:58:28,004 - root - INFO - Couchbase started 2022-03-08 03:58:28,006 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 03:58:28,107 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 03:58:28,378 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:28,684 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:28,693 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 03:58:28,794 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 03:58:29,059 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:29,363 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:29,426 - root - INFO - 172.23.122.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:29,531 - root - INFO - Running systemd command on this server 2022-03-08 03:58:29,531 - root - INFO - running command.raw on 172.23.122.123: systemctl stop couchbase-server.service 2022-03-08 03:58:31,036 - root - INFO - command executed successfully with root 2022-03-08 03:58:31,037 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:31,078 - root - INFO - Couchbase stopped 2022-03-08 03:58:31,079 - root - INFO - running command.raw on 172.23.122.123: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:58:31,094 - root - INFO - command executed successfully with root 2022-03-08 03:58:31,094 - root - INFO - running command.raw on 172.23.122.123: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:58:31,150 - root - INFO - command executed successfully with root 2022-03-08 03:58:31,152 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 03:58:31,254 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 03:58:31,524 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:31,826 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:31,888 - root - INFO - 172.23.122.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:31,888 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:31,967 - root - INFO - Starting couchbase server 2022-03-08 03:58:32,071 - root - INFO - Running systemd command on this server 2022-03-08 03:58:32,071 - root - INFO - running command.raw on 172.23.122.123: systemctl start couchbase-server.service 2022-03-08 03:58:32,093 - root - INFO - command executed successfully with root 2022-03-08 03:58:32,094 - root - INFO - 172.23.122.123:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:58:37,099 - root - INFO - running command.raw on 172.23.122.123: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:58:37,139 - root - INFO - command executed successfully with root 2022-03-08 03:58:37,140 - root - INFO - Couchbase server status: [' Process: 26005 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:58:37,140 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:37,228 - root - INFO - process beam.smp is running on 172.23.122.123: with pid 26156 2022-03-08 03:58:37,229 - root - INFO - Couchbase started 2022-03-08 03:58:37,230 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 03:58:37,356 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 03:58:37,701 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:38,101 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:38,122 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 03:58:38,285 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 03:58:38,630 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:39,005 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:39,069 - root - INFO - 172.23.122.119 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:39,136 - root - INFO - Running systemd command on this server 2022-03-08 03:58:39,136 - root - INFO - running command.raw on 172.23.122.119: systemctl stop couchbase-server.service 2022-03-08 03:58:40,658 - root - INFO - command executed successfully with root 2022-03-08 03:58:40,658 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:40,699 - root - INFO - Couchbase stopped 2022-03-08 03:58:40,699 - root - INFO - running command.raw on 172.23.122.119: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:58:40,716 - root - INFO - command executed successfully with root 2022-03-08 03:58:40,717 - root - INFO - running command.raw on 172.23.122.119: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:58:40,771 - root - INFO - command executed successfully with root 2022-03-08 03:58:40,773 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 03:58:40,870 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 03:58:41,141 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:41,455 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:41,519 - root - INFO - 172.23.122.119 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:41,520 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:41,598 - root - INFO - Starting couchbase server 2022-03-08 03:58:41,705 - root - INFO - Running systemd command on this server 2022-03-08 03:58:41,705 - root - INFO - running command.raw on 172.23.122.119: systemctl start couchbase-server.service 2022-03-08 03:58:41,729 - root - INFO - command executed successfully with root 2022-03-08 03:58:41,729 - root - INFO - 172.23.122.119:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:58:46,734 - root - INFO - running command.raw on 172.23.122.119: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:58:46,765 - root - INFO - command executed successfully with root 2022-03-08 03:58:46,765 - root - INFO - Couchbase server status: [' Process: 120487 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:58:46,765 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:46,837 - root - INFO - process beam.smp is running on 172.23.122.119: with pid 120637 2022-03-08 03:58:46,838 - root - INFO - Couchbase started 2022-03-08 03:58:46,839 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 03:58:46,945 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 03:58:47,263 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:47,573 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:47,582 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 03:58:47,682 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 03:58:47,971 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:48,281 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:48,346 - root - INFO - 172.23.122.120 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:48,413 - root - INFO - Running systemd command on this server 2022-03-08 03:58:48,413 - root - INFO - running command.raw on 172.23.122.120: systemctl stop couchbase-server.service 2022-03-08 03:58:49,922 - root - INFO - command executed successfully with root 2022-03-08 03:58:49,922 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:49,965 - root - INFO - Couchbase stopped 2022-03-08 03:58:49,965 - root - INFO - running command.raw on 172.23.122.120: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:58:49,983 - root - INFO - command executed successfully with root 2022-03-08 03:58:49,984 - root - INFO - running command.raw on 172.23.122.120: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:58:50,039 - root - INFO - command executed successfully with root 2022-03-08 03:58:50,041 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 03:58:50,145 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 03:58:50,444 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:50,754 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:50,779 - root - INFO - 172.23.122.120 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:50,779 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:50,820 - root - INFO - Starting couchbase server 2022-03-08 03:58:50,925 - root - INFO - Running systemd command on this server 2022-03-08 03:58:50,925 - root - INFO - running command.raw on 172.23.122.120: systemctl start couchbase-server.service 2022-03-08 03:58:50,949 - root - INFO - command executed successfully with root 2022-03-08 03:58:50,949 - root - INFO - 172.23.122.120:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:58:55,954 - root - INFO - running command.raw on 172.23.122.120: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:58:55,985 - root - INFO - command executed successfully with root 2022-03-08 03:58:55,985 - root - INFO - Couchbase server status: [' Process: 44419 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:58:55,985 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:56,053 - root - INFO - process beam.smp is running on 172.23.122.120: with pid 44569 2022-03-08 03:58:56,053 - root - INFO - Couchbase started 2022-03-08 03:58:56,055 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 03:58:56,158 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 03:58:56,422 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:56,727 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:56,736 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 03:58:56,834 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 03:58:57,095 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:57,395 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:57,458 - root - INFO - 172.23.122.121 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:57,564 - root - INFO - Running systemd command on this server 2022-03-08 03:58:57,565 - root - INFO - running command.raw on 172.23.122.121: systemctl stop couchbase-server.service 2022-03-08 03:58:59,086 - root - INFO - command executed successfully with root 2022-03-08 03:58:59,086 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:59,124 - root - INFO - Couchbase stopped 2022-03-08 03:58:59,125 - root - INFO - running command.raw on 172.23.122.121: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:58:59,142 - root - INFO - command executed successfully with root 2022-03-08 03:58:59,142 - root - INFO - running command.raw on 172.23.122.121: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:58:59,197 - root - INFO - command executed successfully with root 2022-03-08 03:58:59,199 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 03:58:59,300 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 03:58:59,568 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:58:59,877 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:58:59,940 - root - INFO - 172.23.122.121 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:58:59,940 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:58:59,980 - root - INFO - Starting couchbase server 2022-03-08 03:59:00,084 - root - INFO - Running systemd command on this server 2022-03-08 03:59:00,085 - root - INFO - running command.raw on 172.23.122.121: systemctl start couchbase-server.service 2022-03-08 03:59:00,108 - root - INFO - command executed successfully with root 2022-03-08 03:59:00,108 - root - INFO - 172.23.122.121:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:59:05,113 - root - INFO - running command.raw on 172.23.122.121: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:59:05,159 - root - INFO - command executed successfully with root 2022-03-08 03:59:05,159 - root - INFO - Couchbase server status: [' Process: 103954 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:59:05,160 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:59:05,252 - root - INFO - process beam.smp is running on 172.23.122.121: with pid 104104 2022-03-08 03:59:05,252 - root - INFO - Couchbase started 2022-03-08 03:59:05,255 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 03:59:05,431 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 03:59:05,772 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:06,167 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:06,191 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 03:59:06,314 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 03:59:06,655 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:07,062 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:07,125 - root - INFO - 172.23.122.124 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:59:07,302 - root - INFO - Running systemd command on this server 2022-03-08 03:59:07,302 - root - INFO - running command.raw on 172.23.122.124: systemctl stop couchbase-server.service 2022-03-08 03:59:08,846 - root - INFO - command executed successfully with root 2022-03-08 03:59:08,846 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:59:08,932 - root - INFO - Couchbase stopped 2022-03-08 03:59:08,933 - root - INFO - running command.raw on 172.23.122.124: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2022-03-08 03:59:08,953 - root - INFO - command executed successfully with root 2022-03-08 03:59:08,954 - root - INFO - running command.raw on 172.23.122.124: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2022-03-08 03:59:09,016 - root - INFO - command executed successfully with root 2022-03-08 03:59:09,019 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 03:59:09,131 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 03:59:09,401 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:09,711 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:09,774 - root - INFO - 172.23.122.124 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 03:59:09,774 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:59:09,814 - root - INFO - Starting couchbase server 2022-03-08 03:59:09,925 - root - INFO - Running systemd command on this server 2022-03-08 03:59:09,925 - root - INFO - running command.raw on 172.23.122.124: systemctl start couchbase-server.service 2022-03-08 03:59:09,950 - root - INFO - command executed successfully with root 2022-03-08 03:59:09,950 - root - INFO - 172.23.122.124:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 03:59:14,956 - root - INFO - running command.raw on 172.23.122.124: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 03:59:14,985 - root - INFO - command executed successfully with root 2022-03-08 03:59:14,985 - root - INFO - Couchbase server status: [' Process: 97447 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2022-03-08 03:59:14,985 - root - INFO - Checking for process beam.smp on linux 2022-03-08 03:59:15,058 - root - INFO - process beam.smp is running on 172.23.122.124: with pid 97598 2022-03-08 03:59:15,058 - root - INFO - Couchbase started 2022-03-08 03:59:18,335 - root - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:18,339 - root - INFO - {'uptime': '60', 'memoryTotal': 4201676800, 'memoryFree': 3680391168, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:18,341 - root - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:18,345 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:18,351 - root - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv', 'n1ql']) 2022-03-08 03:59:18,351 - root - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv%2Cn1ql 2022-03-08 03:59:18,359 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:18,359 - root - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:18,398 - root - INFO - --> status:True 2022-03-08 03:59:18,400 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 03:59:18,524 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 03:59:18,820 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:19,128 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:19,129 - root - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:19,196 - root - INFO - command executed successfully with root 2022-03-08 03:59:19,196 - root - INFO - ['ok'] 2022-03-08 03:59:19,199 - root - INFO - /diag/eval status on 172.23.123.61:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:19,201 - root - INFO - /diag/eval status on 172.23.123.61:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:19,205 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:19,223 - root - INFO - server: ip:172.23.122.122 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:19,226 - root - INFO - {'uptime': '52', 'memoryTotal': 4201684992, 'memoryFree': 3644375040, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.122:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.122.122', 'ip': '172.23.122.122', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:19,228 - root - ERROR - GET http://172.23.122.122:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:19,228 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:19,234 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:19,234 - root - INFO - settings/web params on 172.23.122.122:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:19,278 - root - INFO - --> status:True 2022-03-08 03:59:19,280 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 03:59:19,392 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 03:59:19,687 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:20,001 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:20,001 - root - INFO - running command.raw on 172.23.122.122: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:20,069 - root - INFO - command executed successfully with root 2022-03-08 03:59:20,069 - root - INFO - ['ok'] 2022-03-08 03:59:20,072 - root - INFO - /diag/eval status on 172.23.122.122:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:20,074 - root - INFO - /diag/eval status on 172.23.122.122:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:20,079 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:20,098 - root - INFO - server: ip:172.23.122.123 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:20,101 - root - INFO - {'uptime': '43', 'memoryTotal': 4201684992, 'memoryFree': 3640758272, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.123:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.122.123', 'ip': '172.23.122.123', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:20,103 - root - ERROR - GET http://172.23.122.123:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:20,104 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:20,110 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:20,110 - root - INFO - settings/web params on 172.23.122.123:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:20,149 - root - INFO - --> status:True 2022-03-08 03:59:20,152 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 03:59:20,258 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 03:59:20,547 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:20,860 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:20,862 - root - INFO - running command.raw on 172.23.122.123: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:20,931 - root - INFO - command executed successfully with root 2022-03-08 03:59:20,931 - root - INFO - ['ok'] 2022-03-08 03:59:20,934 - root - INFO - /diag/eval status on 172.23.122.123:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:20,936 - root - INFO - /diag/eval status on 172.23.122.123:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:20,941 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:20,959 - root - INFO - server: ip:172.23.122.119 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:20,963 - root - INFO - {'uptime': '33', 'memoryTotal': 4201684992, 'memoryFree': 3683278848, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.119:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.122.119', 'ip': '172.23.122.119', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:20,965 - root - ERROR - GET http://172.23.122.119:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:20,965 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:20,971 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:20,971 - root - INFO - settings/web params on 172.23.122.119:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:21,007 - root - INFO - --> status:True 2022-03-08 03:59:21,009 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 03:59:21,114 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 03:59:21,396 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:21,707 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:21,708 - root - INFO - running command.raw on 172.23.122.119: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:21,776 - root - INFO - command executed successfully with root 2022-03-08 03:59:21,776 - root - INFO - ['ok'] 2022-03-08 03:59:21,780 - root - INFO - /diag/eval status on 172.23.122.119:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:21,782 - root - INFO - /diag/eval status on 172.23.122.119:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:21,787 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:21,806 - root - INFO - server: ip:172.23.122.120 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:21,809 - root - INFO - {'uptime': '28', 'memoryTotal': 4201684992, 'memoryFree': 3680694272, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.120:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.122.120', 'ip': '172.23.122.120', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:21,812 - root - ERROR - GET http://172.23.122.120:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:21,812 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:21,818 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:21,818 - root - INFO - settings/web params on 172.23.122.120:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:21,860 - root - INFO - --> status:True 2022-03-08 03:59:21,862 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 03:59:21,970 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 03:59:22,277 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:22,583 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:22,584 - root - INFO - running command.raw on 172.23.122.120: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:22,650 - root - INFO - command executed successfully with root 2022-03-08 03:59:22,650 - root - INFO - ['ok'] 2022-03-08 03:59:22,654 - root - INFO - /diag/eval status on 172.23.122.120:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:22,657 - root - INFO - /diag/eval status on 172.23.122.120:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:22,661 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:22,679 - root - INFO - server: ip:172.23.122.121 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:22,682 - root - INFO - {'uptime': '18', 'memoryTotal': 4201684992, 'memoryFree': 3653935104, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.121:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.122.121', 'ip': '172.23.122.121', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:22,684 - root - ERROR - GET http://172.23.122.121:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:22,684 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:22,690 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:22,690 - root - INFO - settings/web params on 172.23.122.121:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:22,729 - root - INFO - --> status:True 2022-03-08 03:59:22,730 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 03:59:22,840 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 03:59:23,118 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:23,429 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:23,430 - root - INFO - running command.raw on 172.23.122.121: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:23,497 - root - INFO - command executed successfully with root 2022-03-08 03:59:23,498 - root - INFO - ['ok'] 2022-03-08 03:59:23,501 - root - INFO - /diag/eval status on 172.23.122.121:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:23,504 - root - INFO - /diag/eval status on 172.23.122.121:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:23,509 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:23,527 - root - INFO - server: ip:172.23.122.124 port:8091 ssh_username:root, nodes/self 2022-03-08 03:59:23,530 - root - INFO - {'uptime': '8', 'memoryTotal': 4201689088, 'memoryFree': 3619553280, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.124:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10080-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.122.124', 'ip': '172.23.122.124', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2022-03-08 03:59:23,533 - root - ERROR - GET http://172.23.122.124:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 03:59:23,533 - root - INFO - pools/default params : memoryQuota=2147 2022-03-08 03:59:23,539 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 03:59:23,539 - root - INFO - settings/web params on 172.23.122.124:8091:port=8091&username=Administrator&password=password 2022-03-08 03:59:23,576 - root - INFO - --> status:True 2022-03-08 03:59:23,578 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 03:59:23,688 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 03:59:23,990 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 03:59:24,298 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 03:59:24,299 - root - INFO - running command.raw on 172.23.122.124: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 03:59:24,367 - root - INFO - command executed successfully with root 2022-03-08 03:59:24,367 - root - INFO - ['ok'] 2022-03-08 03:59:24,370 - root - INFO - /diag/eval status on 172.23.122.124:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:24,372 - root - INFO - /diag/eval status on 172.23.122.124:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-03-08 03:59:24,376 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-08 03:59:24,392 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** 2022-03-08 03:59:24,403 - root - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-08 03:59:24,404 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2022-03-08 03:59:24,445 - root - INFO - sleep for 5 secs. ... 2022-03-08 03:59:29,450 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2022-03-08 03:59:29,460 - root - INFO - done initializing cluster 2022-03-08 03:59:30,398 - root - INFO - adding node 172.23.122.122:8091 to cluster 2022-03-08 03:59:30,405 - root - INFO - adding remote node @172.23.122.122:18091 to this cluster @172.23.123.61:8091 2022-03-08 03:59:40,417 - root - INFO - rebalance progress took 10.01 seconds 2022-03-08 03:59:40,417 - root - INFO - sleep for 10 seconds after rebalance... 2022-03-08 03:59:52,222 - root - INFO - adding node 172.23.122.123:8091 to cluster 2022-03-08 03:59:52,229 - root - INFO - adding remote node @172.23.122.123:18091 to this cluster @172.23.123.61:8091 2022-03-08 04:00:02,236 - root - INFO - rebalance progress took 10.01 seconds 2022-03-08 04:00:02,237 - root - INFO - sleep for 10 seconds after rebalance... 2022-03-08 04:00:13,979 - root - INFO - adding node 172.23.122.119:8091 to cluster 2022-03-08 04:00:13,987 - root - INFO - adding remote node @172.23.122.119:18091 to this cluster @172.23.123.61:8091 2022-03-08 04:00:24,003 - root - INFO - rebalance progress took 10.01 seconds 2022-03-08 04:00:24,003 - root - INFO - sleep for 10 seconds after rebalance... 2022-03-08 04:00:35,880 - root - INFO - Node 172.23.122.119 not part of cluster inactiveAdded 2022-03-08 04:00:35,881 - root - INFO - Node 172.23.122.122 not part of cluster inactiveAdded 2022-03-08 04:00:35,881 - root - INFO - Node 172.23.122.123 not part of cluster inactiveAdded 2022-03-08 04:00:35,927 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.119,ns_1@172.23.122.122,ns_1@172.23.122.123,ns_1@172.23.123.61', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-08 04:00:35,950 - root - INFO - rebalance operation started 2022-03-08 04:00:45,965 - root - INFO - Rebalance - status: none, progress: 100.00% 2022-03-08 04:00:45,969 - root - INFO - rebalancing was completed with progress: 100% in 10.018765449523926 sec 2022-03-08 04:00:45,972 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:00:46,079 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:00:46,368 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:46,683 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:46,684 - root - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-08 04:00:46,753 - root - INFO - command executed successfully with root 2022-03-08 04:00:46,753 - root - INFO - ['ok'] 2022-03-08 04:00:46,753 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 2022-03-08 04:00:46,988 - root - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=standard_bucket0&ramQuotaMB=2147&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-08 04:00:46,994 - root - INFO - 0.01 seconds to create bucket standard_bucket0 2022-03-08 04:00:46,994 - root - INFO - waiting for memcached bucket : standard_bucket0 in 172.23.123.61 to accept set ops 2022-03-08 04:00:47,603 - root - INFO - creating direct client 172.23.122.122:11210 standard_bucket0 2022-03-08 04:00:47,656 - root - INFO - creating direct client 172.23.123.61:11210 standard_bucket0 2022-03-08 04:00:47,758 - root - INFO - creating direct client 172.23.122.122:11210 standard_bucket0 2022-03-08 04:00:47,808 - root - INFO - creating direct client 172.23.123.61:11210 standard_bucket0 2022-03-08 04:00:48,795 - root - INFO - bucket 'standard_bucket0' was created with per node RAM quota: 2147 2022-03-08 04:00:48,796 - root - INFO - ============== basetestcase setup was finished for test #1 test_online_upgrade ============== 2022-03-08 04:00:48,805 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:00:48,916 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:00:49,201 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:49,510 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:49,512 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:00:49,614 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:00:49,901 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:50,211 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:50,214 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:00:50,323 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:00:50,595 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:50,900 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:50,903 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:00:51,008 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:00:51,283 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:51,595 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:51,598 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:00:51,700 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:00:51,996 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:52,302 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:52,306 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:00:52,408 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:00:52,684 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:52,997 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:52,999 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:00:53,108 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:00:53,388 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:53,699 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:00:58,704 - root - INFO - ------- Cluster statistics ------- 2022-03-08 04:00:58,704 - root - INFO - 172.23.122.119:8091 => {'services': ['index'], 'cpu_utilization': 29.07268170426065, 'mem_free': 3611242496, 'mem_total': 4201684992, 'swap_mem_used': 76099584, 'swap_mem_total': 3758092288} 2022-03-08 04:00:58,704 - root - INFO - 172.23.122.122:8091 => {'services': ['kv'], 'cpu_utilization': 1.758793969849246, 'mem_free': 3559358464, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-03-08 04:00:58,705 - root - INFO - 172.23.122.123:8091 => {'services': ['index'], 'cpu_utilization': 8.816120906801007, 'mem_free': 3563782144, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-03-08 04:00:58,705 - root - INFO - 172.23.123.61:8091 => {'services': ['kv', 'n1ql'], 'cpu_utilization': 1.2531328320802, 'mem_free': 3537973248, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-03-08 04:00:58,705 - root - INFO - --- End of cluster statistics --- 2022-03-08 04:00:58,705 - root - INFO - Time to execute basesetup : 176.84698343276978 2022-03-08 04:00:58,706 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:00:58,812 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:00:59,099 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:00:59,418 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:01:01,459 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 04:01:01,467 - root - INFO - cannot find service node eventing in cluster 2022-03-08 04:01:01,475 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 04:01:01,479 - root - INFO - ============== EventingBaseTest setup has completed ============== 2022-03-08 04:01:01,480 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:01:01,585 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:01:01,888 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:01:02,216 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:01:04,214 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 04:01:04,215 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:01:04,321 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:01:04,613 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:01:04,934 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:01:04,978 - root - INFO - Initial status of 172.23.123.61 cluster is healthy 2022-03-08 04:01:04,979 - root - INFO - current status of 172.23.123.61 is healthy 2022-03-08 04:01:04,991 - root - INFO - Initial status of 172.23.122.122 cluster is healthy 2022-03-08 04:01:04,991 - root - INFO - current status of 172.23.122.122 is healthy 2022-03-08 04:01:05,002 - root - INFO - Initial status of 172.23.122.123 cluster is healthy 2022-03-08 04:01:05,002 - root - INFO - current status of 172.23.122.123 is healthy 2022-03-08 04:01:05,011 - root - INFO - Initial status of 172.23.122.119 cluster is healthy 2022-03-08 04:01:05,012 - root - INFO - current status of 172.23.122.119 is healthy 2022-03-08 04:01:05,019 - root - INFO - Initial status of 172.23.122.120 cluster is healthy 2022-03-08 04:01:05,019 - root - INFO - current status of 172.23.122.120 is healthy 2022-03-08 04:01:05,026 - root - INFO - Initial status of 172.23.122.121 cluster is healthy 2022-03-08 04:01:05,027 - root - INFO - current status of 172.23.122.121 is healthy 2022-03-08 04:01:05,033 - root - INFO - Initial status of 172.23.122.124 cluster is healthy 2022-03-08 04:01:05,033 - root - INFO - current status of 172.23.122.124 is healthy 2022-03-08 04:01:05,041 - root - INFO - list of index nodes in cluster: [ip:172.23.122.119 port:8091 ssh_username:root, ip:172.23.122.123 port:8091 ssh_username:root] 2022-03-08 04:01:05,056 - root - INFO - {'indexer.settings.storage_mode': 'plasma'} set 2022-03-08 04:01:05,056 - root - INFO - Allowing the indexer to complete restart after setting the internal settings 2022-03-08 04:01:05,056 - root - INFO - sleep for 5 secs. ... 2022-03-08 04:01:10,071 - root - INFO - {'indexer.api.enableTestServer': True} set 2022-03-08 04:01:10,073 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:01:10,179 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:01:10,469 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:01:10,776 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:01:13,202 - root - INFO - create 20160.0 to standard_bucket0 documents... 2022-03-08 04:01:13,246 - root - INFO - creating direct client 172.23.122.122:11210 standard_bucket0 2022-03-08 04:01:13,306 - root - INFO - creating direct client 172.23.123.61:11210 standard_bucket0 ./lib/log_scanner.py:14: DeprecationWarning: invalid escape sequence \s "Basic\s[a-zA-Z]\{10,\}==", ./lib/log_scanner.py:15: DeprecationWarning: invalid escape sequence \[ "Menelaus-Auth-User:\[", 2022-03-08 04:01:30,352 - root - INFO - LOAD IS FINISHED 2022-03-08 04:01:30,362 - root - INFO - {'update': {'start': 0, 'end': 5}, 'remaining': {'start': 5, 'end': 10}} 2022-03-08 04:01:33,152 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 04:01:33,153 - root - INFO - ip:172.23.123.61 port:8091 ssh_username:root 2022-03-08 04:01:33,153 - root - INFO - sleep for 30 secs. ... 2022-03-08 04:02:03,176 - root - INFO - Check if index existed in standard_bucket0 on server 172.23.123.61 2022-03-08 04:02:03,183 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2022-03-08 04:02:03,188 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2022-03-08 04:02:03,312 - root - INFO - TOTAL ELAPSED TIME: 120.36073ms 2022-03-08 04:02:03,312 - root - ERROR - Fail to get index list. List output: {'requestID': '33389f28-6125-42b8-9c8a-69267800cb23', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '120.36073ms', 'executionTime': '120.199899ms', 'resultCount': 0, 'resultSize': 0}} 2022-03-08 04:02:03,312 - root - INFO - Create primary index 2022-03-08 04:02:03,317 - root - INFO - RUN QUERY CREATE PRIMARY INDEX ON standard_bucket0 USING GSI 2022-03-08 04:02:03,322 - root - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket0++USING+GSI 2022-03-08 04:02:06,327 - root - INFO - TOTAL ELAPSED TIME: 3.002895677s 2022-03-08 04:02:06,327 - root - INFO - Check if index is online 2022-03-08 04:02:06,332 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2022-03-08 04:02:06,337 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2022-03-08 04:02:06,350 - root - INFO - TOTAL ELAPSED TIME: 11.382817ms 2022-03-08 04:02:06,361 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:02:06,469 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:02:06,749 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:02:07,060 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:02:07,083 - root - INFO - list of index nodes in cluster: [ip:172.23.122.119 port:8091 ssh_username:root, ip:172.23.122.123 port:8091 ssh_username:root] 2022-03-08 04:02:07,088 - root - INFO - Making a rest request api=http://172.23.122.119:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-03-08 04:02:07,103 - root - INFO - {'queryport.client.waitForScheduledIndex': False} set 2022-03-08 04:02:07,104 - root - INFO - Making a rest request api=http://172.23.122.119:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-03-08 04:02:07,116 - root - INFO - {'indexer.allowScheduleCreateRebal': True} set 2022-03-08 04:02:08,082 - root - INFO - RUN QUERY CREATE INDEX `index_name_0` ON standard_bucket0(job_title) USING GSI WITH {'defer_build': True} 2022-03-08 04:02:08,086 - root - INFO - query params : statement=CREATE+INDEX+%60index_name_0%60+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D 2022-03-08 04:02:09,001 - root - INFO - TOTAL ELAPSED TIME: 911.960775ms 2022-03-08 04:02:09,001 - root - INFO - BUILD INDEX on standard_bucket0(`index_name_0`) USING GSI 2022-03-08 04:02:10,008 - root - INFO - RUN QUERY BUILD INDEX on standard_bucket0(`index_name_0`) USING GSI 2022-03-08 04:02:10,014 - root - INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28%60index_name_0%60%29+USING+GSI 2022-03-08 04:02:10,068 - root - INFO - TOTAL ELAPSED TIME: 51.652449ms 2022-03-08 04:02:10,073 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-03-08 04:02:10,078 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-03-08 04:02:10,094 - root - INFO - TOTAL ELAPSED TIME: 12.854545ms 2022-03-08 04:02:11,101 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-03-08 04:02:11,105 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-03-08 04:02:11,124 - root - INFO - TOTAL ELAPSED TIME: 16.40092ms 2022-03-08 04:02:12,131 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-03-08 04:02:12,136 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-03-08 04:02:12,152 - root - INFO - TOTAL ELAPSED TIME: 14.437209ms 2022-03-08 04:02:13,154 - root - INFO - === BEFORE: ['create_index'] === 2022-03-08 04:02:13,154 - root - INFO - [ip:172.23.122.119 port:8091 ssh_username:root] 2022-03-08 04:02:14,078 - root - INFO - RUN QUERY CREATE INDEX `employee2584caf24e2b4bc9a31c0eab2752d307job_title` ON standard_bucket0(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'nodes': ['172.23.122.119:8091'], 'defer_build': True} 2022-03-08 04:02:14,083 - root - INFO - query params : statement=CREATE+INDEX+%60employee2584caf24e2b4bc9a31c0eab2752d307job_title%60+ON+standard_bucket0%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.122.119%3A8091%27%5D%2C+%27defer_build%27%3A+True%7D 2022-03-08 04:02:14,143 - root - INFO - TOTAL ELAPSED TIME: 56.788971ms 2022-03-08 04:02:14,148 - root - INFO - RUN QUERY CREATE INDEX `employee2584caf24e2b4bc9a31c0eab2752d307join_yr` ON standard_bucket0(join_yr) WHERE join_yr > 2010 and join_yr < 2014 USING GSI WITH {'nodes': ['172.23.122.119:8091'], 'defer_build': True} 2022-03-08 04:02:14,153 - root - INFO - query params : statement=CREATE+INDEX+%60employee2584caf24e2b4bc9a31c0eab2752d307join_yr%60+ON+standard_bucket0%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.122.119%3A8091%27%5D%2C+%27defer_build%27%3A+True%7D 2022-03-08 04:02:14,225 - root - INFO - TOTAL ELAPSED TIME: 69.558198ms 2022-03-08 04:02:14,226 - root - INFO - BUILD INDEX on standard_bucket0(employee2584caf24e2b4bc9a31c0eab2752d307job_title,employee2584caf24e2b4bc9a31c0eab2752d307join_yr) USING GSI 2022-03-08 04:02:15,232 - root - INFO - RUN QUERY BUILD INDEX on standard_bucket0(employee2584caf24e2b4bc9a31c0eab2752d307job_title,employee2584caf24e2b4bc9a31c0eab2752d307join_yr) USING GSI 2022-03-08 04:02:15,237 - root - INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28employee2584caf24e2b4bc9a31c0eab2752d307job_title%2Cemployee2584caf24e2b4bc9a31c0eab2752d307join_yr%29+USING+GSI 2022-03-08 04:02:15,261 - root - INFO - TOTAL ELAPSED TIME: 22.212891ms 2022-03-08 04:02:16,268 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2584caf24e2b4bc9a31c0eab2752d307job_title' 2022-03-08 04:02:16,273 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2584caf24e2b4bc9a31c0eab2752d307job_title%27 2022-03-08 04:02:16,297 - root - INFO - TOTAL ELAPSED TIME: 19.480273ms 2022-03-08 04:02:17,304 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2584caf24e2b4bc9a31c0eab2752d307job_title' 2022-03-08 04:02:17,310 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2584caf24e2b4bc9a31c0eab2752d307job_title%27 2022-03-08 04:02:17,331 - root - INFO - TOTAL ELAPSED TIME: 17.707001ms 2022-03-08 04:02:18,337 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2584caf24e2b4bc9a31c0eab2752d307job_title' 2022-03-08 04:02:18,341 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2584caf24e2b4bc9a31c0eab2752d307job_title%27 2022-03-08 04:02:18,358 - root - INFO - TOTAL ELAPSED TIME: 14.176426ms 2022-03-08 04:02:18,363 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2584caf24e2b4bc9a31c0eab2752d307join_yr' 2022-03-08 04:02:18,369 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2584caf24e2b4bc9a31c0eab2752d307join_yr%27 2022-03-08 04:02:18,383 - root - INFO - TOTAL ELAPSED TIME: 10.868505ms 2022-03-08 04:02:18,383 - root - INFO - === IN_BETWEEN: ['query', 'query_with_explain'] === 2022-03-08 04:02:18,384 - root - INFO - Query : SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id 2022-03-08 04:02:18,385 - root - INFO - FROM clause ===== is standard_bucket0 where job_title IS NOT NULL 2022-03-08 04:02:18,385 - root - INFO - WHERE clause ===== is None 2022-03-08 04:02:18,386 - root - INFO - UNNEST clause ===== is None 2022-03-08 04:02:18,386 - root - INFO - SELECT clause ===== is {"*" : doc,} 2022-03-08 04:02:18,387 - root - INFO - ORDER clause ========= is doc["job_title"],doc["_id"], 2022-03-08 04:02:18,387 - root - INFO - ORDER clause ========= is doc["job_title"],doc["_id"], 2022-03-08 04:02:18,388 - root - INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"],"job_title" : doc["job_title"]}; where_clause=None 2022-03-08 04:02:18,965 - root - INFO - ORDER clause ========= is doc["job_title"],doc["_id"], 2022-03-08 04:02:19,390 - root - INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id >>>>>> 2022-03-08 04:02:19,418 - root - INFO - RUN QUERY EXPLAIN SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id 2022-03-08 04:02:19,440 - root - INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+standard_bucket0+where+job_title+IS+NOT+NULL+ORDER+BY+job_title%2C_id 2022-03-08 04:02:19,466 - root - INFO - TOTAL ELAPSED TIME: 3.21286ms 2022-03-08 04:02:19,466 - root - INFO - {'requestID': 'a7f97bb4-8fc3-43a0-bf77-ce9432bd3e26', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2584caf24e2b4bc9a31c0eab2752d307job_title', 'index_id': 'cac71904ad50f29f', 'index_projection': {'primary_key': True}, 'keyspace': 'standard_bucket0', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'inclusion': 0, 'low': 'null'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'standard_bucket0', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`standard_bucket0`.`job_title`) is not null)'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`standard_bucket0`.`job_title`)'}, {'expr': '(`standard_bucket0`.`_id`)'}]}, {'#operator': 'FinalProject'}]}, 'text': 'SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.21286ms', 'executionTime': '3.149851ms', 'resultCount': 1, 'resultSize': 919}} 2022-03-08 04:02:19,466 - root - INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id >>>>>> 2022-03-08 04:02:19,466 - root - INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id >>>>>> 2022-03-08 04:02:19,491 - root - INFO - RUN QUERY EXPLAIN SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id 2022-03-08 04:02:19,514 - root - INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+standard_bucket0+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+ 2022-03-08 04:02:19,547 - root - INFO - TOTAL ELAPSED TIME: 2.350856ms 2022-03-08 04:02:19,552 - root - INFO - {'requestID': 'd5f8794b-e657-4314-a916-d0cff7f5594e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2584caf24e2b4bc9a31c0eab2752d307join_yr', 'index_id': '12ade6dc5b192e64', 'index_projection': {'primary_key': True}, 'keyspace': 'standard_bucket0', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'standard_bucket0', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`standard_bucket0`.`join_yr`)) and ((`standard_bucket0`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`standard_bucket0`.`_id`)'}]}, {'#operator': 'FinalProject'}]}, 'text': 'SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.350856ms', 'executionTime': '2.274253ms', 'resultCount': 1, 'resultSize': 949}} 2022-03-08 04:02:19,563 - root - INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id >>>>>> 2022-03-08 04:02:19,574 - root - INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id >>>>>> 2022-03-08 04:02:19,580 - root - INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id >>>>>> 2022-03-08 04:02:19,634 - root - INFO - Query : SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id 2022-03-08 04:02:19,635 - root - INFO - FROM clause ===== is standard_bucket0 2022-03-08 04:02:19,635 - root - INFO - WHERE clause ===== is doc["join_yr"] > 2010 and doc["join_yr"] < 2014 2022-03-08 04:02:19,635 - root - INFO - UNNEST clause ===== is None 2022-03-08 04:02:19,635 - root - INFO - SELECT clause ===== is {"*" : doc,} 2022-03-08 04:02:19,636 - root - INFO - ORDER clause ========= is doc["_id"], 2022-03-08 04:02:19,636 - root - INFO - ORDER clause ========= is doc["_id"], 2022-03-08 04:02:19,636 - root - INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014 2022-03-08 04:02:19,636 - root - INFO - -->where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014 2022-03-08 04:02:20,237 - root - INFO - ORDER clause ========= is doc["_id"], 2022-03-08 04:02:20,592 - root - INFO - <<<<< START Executing Query SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id >>>>>> 2022-03-08 04:02:20,631 - root - INFO - RUN QUERY SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id 2022-03-08 04:02:20,665 - root - INFO - query params : statement=SELECT+%2A+FROM+standard_bucket0+where+job_title+IS+NOT+NULL+ORDER+BY+job_title%2C_id&scan_consistency=request_plus 2022-03-08 04:02:21,048 - root - INFO - create 20160 to standard_bucket0 documents... 2022-03-08 04:02:21,106 - root - INFO - creating direct client 172.23.122.122:11210 standard_bucket0 2022-03-08 04:02:21,188 - root - INFO - creating direct client 172.23.123.61:11210 standard_bucket0 2022-03-08 04:02:21,317 - gsi.upgrade_gsi - INFO - Upgrading servers to 7.1.0-2434... 2022-03-08 04:02:21,964 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.119,ns_1@172.23.122.122,ns_1@172.23.122.123,ns_1@172.23.123.61', 'ejectedNodes': 'ns_1@172.23.122.119', 'user': 'Administrator', 'password': 'password'} 2022-03-08 04:02:21,970 - root - INFO - rebalance operation started 2022-03-08 04:02:23,044 - root - INFO - TOTAL ELAPSED TIME: 1.931630508s 2022-03-08 04:02:23,044 - root - INFO - Analyzing Actual Result 2022-03-08 04:02:23,061 - root - INFO - Analyzing Expected Result 2022-03-08 04:02:32,015 - root - INFO - Rebalance - status: none, progress: 100.00% 2022-03-08 04:02:32,172 - root - INFO - rebalancing was completed with progress: 100% in 10.20224642753601 sec 2022-03-08 04:02:32,173 - root - INFO - servers ['172.23.122.119'] will be upgraded to 7.1.0-2434 version 2022-03-08 04:02:32,182 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:02:32,388 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:02:33,007 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:02:33,774 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:02:33,780 - root - INFO - finding build 7.1.0-2434 for machine ip:172.23.122.119 port:8091 ssh_username:root 2022-03-08 04:02:33,781 - root - INFO - running command.raw on 172.23.122.119: /sbin/iptables -F 2022-03-08 04:02:33,875 - root - INFO - command executed successfully with root 2022-03-08 04:02:33,892 - root - INFO - running command.raw on 172.23.122.119: /sbin/iptables -t nat -F 2022-03-08 04:02:33,951 - root - INFO - command executed successfully with root 2022-03-08 04:02:33,951 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:02:34,225 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:02:34,524 - root - INFO - running command.raw on 172.23.122.119: cd /tmp;wget -nc -q -O couchbase-server-enterprise-7.1.0-2434-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2434/couchbase-server-enterprise-7.1.0-2434-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:02:34,618 - root - INFO - command executed successfully with root 2022-03-08 04:02:34,619 - root - INFO - total 409M 2022-03-08 04:02:34,619 - root - INFO - -rw-r--r--. 1 root root 409M Mar 7 03:04 couchbase-server-enterprise-7.1.0-2434-centos7.x86_64.rpm 2022-03-08 04:02:34,619 - root - INFO - drwx------. 3 root root 16 Feb 26 08:32 systemd-private-2bb6ab08e5504815b4fc1d8af624077a-ntpd.service-UgCL2B 2022-03-08 04:02:34,840 - root - INFO - deliverable_type : rpm 2022-03-08 04:02:34,840 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2434-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:02:34,845 - root - INFO - running command.raw on 172.23.122.119: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2434-centos7.x86_64.rpm 2022-03-08 04:03:09,592 - root - INFO - <<<<< Done Executing Query SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id >>>>>> 2022-03-08 04:03:09,593 - root - INFO - <<<<< START Executing Query SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id >>>>>> 2022-03-08 04:03:09,599 - root - INFO - RUN QUERY SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id 2022-03-08 04:03:09,604 - root - INFO - query params : statement=SELECT+%2A+FROM+standard_bucket0+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus 2022-03-08 04:03:10,804 - root - INFO - TOTAL ELAPSED TIME: 1.093773331s 2022-03-08 04:03:10,804 - root - INFO - Analyzing Actual Result 2022-03-08 04:03:10,814 - root - INFO - Analyzing Expected Result 2022-03-08 04:03:31,439 - root - INFO - <<<<< Done Executing Query SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id >>>>>> 2022-03-08 04:03:31,440 - root - INFO - <<<<< Done VERIFYING Query SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id >>>>>> 2022-03-08 04:03:31,440 - root - INFO - <<<<< Done VERIFYING Query SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id >>>>>> 2022-03-08 04:03:31,449 - root - INFO - command executed successfully with root 2022-03-08 04:03:31,449 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-03-08 04:03:31,450 - root - INFO - running command.raw on 172.23.122.119: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:03:31,514 - root - INFO - command executed successfully with root 2022-03-08 04:03:31,514 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:03:31,597 - root - INFO - process beam.smp is running on 172.23.122.119: with pid 123304 2022-03-08 04:03:31,756 - root - INFO - Running systemd command on this server 2022-03-08 04:03:31,757 - root - INFO - running command.raw on 172.23.122.119: systemctl stop couchbase-server.service 2022-03-08 04:03:32,603 - root - INFO - command executed successfully with root 2022-03-08 04:03:32,604 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:03:32,637 - root - INFO - Starting couchbase server 2022-03-08 04:03:32,832 - root - INFO - Running systemd command on this server 2022-03-08 04:03:32,833 - root - INFO - running command.raw on 172.23.122.119: systemctl start couchbase-server.service 2022-03-08 04:03:32,853 - root - INFO - command executed successfully with root 2022-03-08 04:03:32,853 - root - INFO - 172.23.122.119:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:03:37,859 - root - INFO - running command.raw on 172.23.122.119: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:03:37,887 - root - INFO - command executed successfully with root 2022-03-08 04:03:37,888 - root - INFO - Couchbase server status: [' Process: 123367 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=1/FAILURE)'] 2022-03-08 04:03:37,888 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:03:37,956 - root - INFO - process beam.smp is running on 172.23.122.119: with pid 123400 2022-03-08 04:03:37,957 - root - INFO - upgrade 172.23.122.119 to version 7.1.0-2434 is completed 2022-03-08 04:03:38,022 - root - INFO - 172.23.122.119 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:03:38,181 - root - INFO - Running systemd start command on this server 2022-03-08 04:03:38,181 - root - INFO - running command.raw on 172.23.122.119: systemctl start couchbase-server.service 2022-03-08 04:03:38,201 - root - INFO - command executed successfully with root 2022-03-08 04:03:38,201 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-03-08 04:04:08,243 - root - INFO - waiting for ns_server @ 172.23.122.119:8091 2022-03-08 04:04:08,243 - root - INFO - -->is_ns_server_running? 2022-03-08 04:04:08,246 - root - INFO - ns_server @ 172.23.122.119:8091 is running 2022-03-08 04:04:08,246 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-08 04:04:08,246 - root - INFO - settings/web params on 172.23.122.119:8091:port=8091&username=Administrator&password=password 2022-03-08 04:04:08,312 - root - INFO - --> status:True 2022-03-08 04:04:08,312 - root - INFO - sleep for 15 secs. ... 2022-03-08 04:04:23,328 - root - INFO - sleep for 10 secs. ... 2022-03-08 04:04:33,338 - gsi.upgrade_gsi - INFO - ==== Upgrade Complete ==== 2022-03-08 04:04:33,339 - root - INFO - sleep for 120 secs. ... 2022-03-08 04:06:33,411 - root - INFO - Node versions in cluster ['7.1.0-2434-enterprise'] 2022-03-08 04:06:33,815 - root - INFO - adding node 172.23.122.119:8091 to cluster 2022-03-08 04:06:33,826 - root - INFO - adding remote node @172.23.122.119:18091 to this cluster @172.23.123.61:8091 2022-03-08 04:06:43,843 - root - INFO - rebalance progress took 10.02 seconds 2022-03-08 04:06:43,843 - root - INFO - sleep for 10 seconds after rebalance... 2022-03-08 04:06:57,182 - root - INFO - Node 172.23.122.119 not part of cluster inactiveAdded 2022-03-08 04:06:57,209 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.119,ns_1@172.23.122.122,ns_1@172.23.122.123,ns_1@172.23.123.61', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-08 04:07:07,279 - root - INFO - rebalance operation started 2022-03-08 04:07:17,291 - root - INFO - rebalance percentage : 37.00 % 2022-03-08 04:07:17,291 - root - INFO - Rebalance - status: running, progress: 37.00% 2022-03-08 04:07:37,323 - root - INFO - rebalance percentage : 37.00 % 2022-03-08 04:07:37,323 - root - INFO - Rebalance - status: running, progress: 37.00% 2022-03-08 04:07:57,354 - root - INFO - rebalance percentage : 37.00 % 2022-03-08 04:07:57,354 - root - INFO - Rebalance - status: running, progress: 37.00% 2022-03-08 04:08:17,387 - root - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2022-03-08 04:08:17,394 - root - INFO - Latest logs from UI on 172.23.123.61: 2022-03-08 04:08:17,394 - root - ERROR - {'node': 'ns_1@172.23.122.122', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1646741214581, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.122.122' saw that node 'ns_1@172.23.122.119' came up. Tags: []", 'serverTime': '2022-03-08T04:06:54.581Z'} 2022-03-08 04:08:17,394 - root - ERROR - {'node': 'ns_1@172.23.122.123', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1646741214374, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.122.123' saw that node 'ns_1@172.23.122.119' came up. Tags: []", 'serverTime': '2022-03-08T04:06:54.374Z'} 2022-03-08 04:08:17,394 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1646741214205, 'shortText': 'message', 'text': 'Conflicting configuration changes to field email_alerts:\n[{recipients,["root@localhost"]},\n {sender,"couchbase@localhost"},\n {enabled,false},\n {email_server,[{user,[]},\n {pass,"*****"},\n {host,"localhost"},\n {port,25},\n {encrypt,false}]},\n {alerts,[auto_failover_node,auto_failover_maximum_reached,\n auto_failover_other_nodes_down,auto_failover_cluster_too_small,\n auto_failover_disabled,ip,disk,overhead,ep_oom_errors,\n ep_item_commit_failed,audit_dropped_events,indexer_ram_max_usage,\n ep_clock_cas_drift_threshold_exceeded,communication_issue]}] and\n[{recipients,["root@localhost"]},\n {sender,"couchbase@localhost"},\n {enabled,false},\n {email_server,[{user,[]},\n {pass,"*****"},\n {host,"localhost"},\n {port,25},\n {encrypt,false}]},\n {alerts,[auto_failover_node,auto_failover_maximum_reached,\n auto_failover_other_nodes_down,auto_failover_cluster_too_small,\n auto_failover_disabled,ip,disk,overhead,ep_oom_errors,\n ep_item_commit_failed,audit_dropped_events,indexer_ram_max_usage,\n ep_clock_cas_drift_threshold_exceeded,communication_issue,\n time_out_of_sync,disk_usage_analyzer_stuck]},\n {pop_up_alerts,[auto_failover_node,auto_failover_maximum_reached,\n auto_failover_other_nodes_down,\n auto_failover_cluster_too_small,auto_failover_disabled,ip,\n disk,overhead,ep_oom_errors,ep_item_commit_failed,\n audit_dropped_events,indexer_ram_max_usage,\n ep_clock_cas_drift_threshold_exceeded,communication_issue,\n time_out_of_sync,disk_usage_analyzer_stuck]}], choosing the former.', 'serverTime': '2022-03-08T04:06:54.205Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1646741214201, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.123.61' saw that node 'ns_1@172.23.122.119' came up. Tags: []", 'serverTime': '2022-03-08T04:06:54.201Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 0, 'module': 'ns_cluster', 'tstamp': 1646741214010, 'shortText': 'message', 'text': "Started node add transaction by adding node 'ns_1@172.23.122.119' to nodes_wanted (group: undefined)", 'serverTime': '2022-03-08T04:06:54.010Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.122.122', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1646740948791, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.122.122' saw that node 'ns_1@172.23.122.119' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-08T04:02:28.791Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1646740948791, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.123.61' saw that node 'ns_1@172.23.122.119' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-08T04:02:28.791Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.122.123', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1646740948790, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.122.123' saw that node 'ns_1@172.23.122.119' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-08T04:02:28.790Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1646740948667, 'shortText': 'message', 'text': 'Rebalance completed successfully.\nRebalance Operation Id = 2408f29bd96efb730dfa04c9edcfe07a', 'serverTime': '2022-03-08T04:02:28.667Z'} 2022-03-08 04:08:17,395 - root - ERROR - {'node': 'ns_1@172.23.122.119', 'type': 'info', 'code': 1, 'module': 'ns_cluster', 'tstamp': 1646740948658, 'shortText': 'message', 'text': "Node 'ns_1@172.23.122.119' is leaving cluster.", 'serverTime': '2022-03-08T04:02:28.658Z'} [, , , , , ] Tue Mar 8 04:08:17 2022 [, , , , , , , , ] 2022-03-08 04:08:17,411 - root - INFO - list of index nodes in cluster: [ip:172.23.122.119 port:8091 ssh_username:root, ip:172.23.122.123 port:8091 ssh_username:root] 2022-03-08 04:08:17,412 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:08:17,511 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:08:17,777 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:18,074 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:18,082 - root - INFO - /diag/eval status on 172.23.122.123:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-08 04:08:18,083 - root - INFO - running command.raw on 172.23.122.123: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2022-03-08 04:08:18,137 - root - INFO - command executed successfully with root 2022-03-08 04:08:18,145 - root - INFO - list of kv nodes in cluster: [ip:172.23.122.122 port:8091 ssh_username:root, ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 04:08:18,146 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:08:18,251 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:08:18,543 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:18,852 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:18,860 - root - INFO - /diag/eval status on 172.23.122.122:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-08 04:08:18,860 - root - INFO - running command.raw on 172.23.122.122: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-03-08 04:08:18,918 - root - INFO - command executed successfully with root 2022-03-08 04:08:18,921 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:08:19,024 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:08:19,297 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:19,603 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:19,611 - root - INFO - /diag/eval status on 172.23.123.61:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2022-03-08 04:08:19,611 - root - INFO - running command.raw on 172.23.123.61: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2022-03-08 04:08:19,669 - root - INFO - command executed successfully with root 2022-03-08 04:08:19,680 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.123.61 port:8091 ssh_username:root] 2022-03-08 04:08:19,680 - root - INFO - CHECK FOR PRIMARY INDEXES 2022-03-08 04:08:19,680 - root - INFO - DROP PRIMARY INDEX ON standard_bucket0 USING GSI 2022-03-08 04:08:19,684 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2022-03-08 04:08:19,688 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2022-03-08 04:08:19,704 - root - INFO - TOTAL ELAPSED TIME: 14.704815ms 2022-03-08 04:08:19,708 - root - INFO - RUN QUERY DROP PRIMARY INDEX ON standard_bucket0 USING GSI 2022-03-08 04:08:19,712 - root - INFO - query params : statement=DROP+PRIMARY+INDEX+ON+standard_bucket0+USING+GSI 2022-03-08 04:08:19,728 - root - INFO - TOTAL ELAPSED TIME: 14.832354ms 2022-03-08 04:08:19,737 - root - INFO - ============== EventingBaseTest tearDown has started ============== 2022-03-08 04:08:19,754 - root - INFO - cannot find service node eventing in cluster 2022-03-08 04:08:19,779 - root - INFO - ------- Cluster statistics ------- 2022-03-08 04:08:19,779 - root - INFO - 172.23.122.119:8091 => {'services': ['index'], 'cpu_utilization': 0.8791760864104496, 'mem_free': 3485872128, 'mem_total': 4201684992, 'swap_mem_used': 76099584, 'swap_mem_total': 3758092288} 2022-03-08 04:08:19,779 - root - INFO - 172.23.122.122:8091 => {'services': ['kv'], 'cpu_utilization': 1.2531328320802, 'mem_free': 3402752000, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-03-08 04:08:19,779 - root - INFO - 172.23.122.123:8091 => {'services': ['index'], 'cpu_utilization': 10.15228426395939, 'mem_free': 3386904576, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-03-08 04:08:19,779 - root - INFO - 172.23.123.61:8091 => {'services': ['kv', 'n1ql'], 'cpu_utilization': 1.515151515151515, 'mem_free': 3222679552, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2022-03-08 04:08:19,780 - root - INFO - --- End of cluster statistics --- 2022-03-08 04:08:19,781 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:08:19,880 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:08:20,158 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:20,463 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:20,466 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:08:20,567 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:08:20,852 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:21,154 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:21,157 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:08:21,260 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:08:21,530 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:21,847 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:21,850 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:08:21,957 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:08:22,227 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:08:22,540 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:08:27,478 - root - INFO - ============== basetestcase cleanup was started for test #1 test_online_upgrade ============== 2022-03-08 04:08:27,507 - root - INFO - deleting existing buckets ['standard_bucket0'] on 172.23.123.61 2022-03-08 04:08:28,632 - root - INFO - waiting for bucket deletion to complete.... 2022-03-08 04:08:28,635 - root - INFO - node 172.23.123.61 existing buckets : [] 2022-03-08 04:08:28,636 - root - INFO - deleted bucket : standard_bucket0 from 172.23.123.61 2022-03-08 04:08:28,643 - root - INFO - Could not find any buckets for node 172.23.122.122, nothing to delete 2022-03-08 04:08:28,657 - root - INFO - Could not find any buckets for node 172.23.122.123, nothing to delete 2022-03-08 04:08:28,664 - root - INFO - Could not find any buckets for node 172.23.122.119, nothing to delete 2022-03-08 04:08:28,666 - root - INFO - Removing user 'clientuser'... 2022-03-08 04:08:28,676 - root - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-08 04:08:28,676 - root - INFO - b'"User was not found."' 2022-03-08 04:08:28,680 - root - INFO - -->is_ns_server_running? 2022-03-08 04:08:28,693 - root - INFO - rebalancing all nodes in order to remove nodes 2022-03-08 04:08:28,695 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.119,ns_1@172.23.122.122,ns_1@172.23.122.123,ns_1@172.23.123.61', 'ejectedNodes': 'ns_1@172.23.122.119,ns_1@172.23.122.122,ns_1@172.23.122.123', 'user': 'Administrator', 'password': 'password'} 2022-03-08 04:08:28,700 - root - INFO - rebalance operation started 2022-03-08 04:08:28,706 - root - INFO - rebalance percentage : 0.00 % 2022-03-08 04:08:38,720 - root - INFO - rebalance percentage : 37.00 % 2022-03-08 04:08:48,747 - root - INFO - rebalance percentage : 37.00 % 2022-03-08 04:08:58,762 - root - INFO - rebalance percentage : 37.00 % 2022-03-08 04:09:08,775 - root - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2022-03-08 04:09:08,781 - root - INFO - Latest logs from UI on 172.23.123.61: 2022-03-08 04:09:08,781 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1646741308694, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@172.23.123.61\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\nStarting rebalance from test, ejected nodes [\'ns_1@172.23.122.119\', \'ns_1@172.23.122.122\', \'ns_1@172.23.122.123\']', 'serverTime': '2022-03-08T04:08:28.694Z'} 2022-03-08 04:09:08,781 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 11, 'module': 'menelaus_web', 'tstamp': 1646741308630, 'shortText': 'message', 'text': 'Deleted bucket "standard_bucket0"\n', 'serverTime': '2022-03-08T04:08:28.630Z'} 2022-03-08 04:09:08,781 - root - ERROR - {'node': 'ns_1@172.23.122.122', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1646741307658, 'shortText': 'message', 'text': 'Shutting down bucket "standard_bucket0" on \'ns_1@172.23.122.122\' for deletion', 'serverTime': '2022-03-08T04:08:27.658Z'} 2022-03-08 04:09:08,782 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1646741307649, 'shortText': 'message', 'text': 'Shutting down bucket "standard_bucket0" on \'ns_1@172.23.123.61\' for deletion', 'serverTime': '2022-03-08T04:08:27.649Z'} 2022-03-08 04:09:08,782 - root - ERROR - {'node': 'ns_1@172.23.122.122', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1646741214581, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.122.122' saw that node 'ns_1@172.23.122.119' came up. Tags: []", 'serverTime': '2022-03-08T04:06:54.581Z'} 2022-03-08 04:09:08,782 - root - ERROR - {'node': 'ns_1@172.23.122.123', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1646741214374, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.122.123' saw that node 'ns_1@172.23.122.119' came up. Tags: []", 'serverTime': '2022-03-08T04:06:54.374Z'} 2022-03-08 04:09:08,782 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1646741214205, 'shortText': 'message', 'text': 'Conflicting configuration changes to field email_alerts:\n[{recipients,["root@localhost"]},\n {sender,"couchbase@localhost"},\n {enabled,false},\n {email_server,[{user,[]},\n {pass,"*****"},\n {host,"localhost"},\n {port,25},\n {encrypt,false}]},\n {alerts,[auto_failover_node,auto_failover_maximum_reached,\n auto_failover_other_nodes_down,auto_failover_cluster_too_small,\n auto_failover_disabled,ip,disk,overhead,ep_oom_errors,\n ep_item_commit_failed,audit_dropped_events,indexer_ram_max_usage,\n ep_clock_cas_drift_threshold_exceeded,communication_issue]}] and\n[{recipients,["root@localhost"]},\n {sender,"couchbase@localhost"},\n {enabled,false},\n {email_server,[{user,[]},\n {pass,"*****"},\n {host,"localhost"},\n {port,25},\n {encrypt,false}]},\n {alerts,[auto_failover_node,auto_failover_maximum_reached,\n auto_failover_other_nodes_down,auto_failover_cluster_too_small,\n auto_failover_disabled,ip,disk,overhead,ep_oom_errors,\n ep_item_commit_failed,audit_dropped_events,indexer_ram_max_usage,\n ep_clock_cas_drift_threshold_exceeded,communication_issue,\n time_out_of_sync,disk_usage_analyzer_stuck]},\n {pop_up_alerts,[auto_failover_node,auto_failover_maximum_reached,\n auto_failover_other_nodes_down,\n auto_failover_cluster_too_small,auto_failover_disabled,ip,\n disk,overhead,ep_oom_errors,ep_item_commit_failed,\n audit_dropped_events,indexer_ram_max_usage,\n ep_clock_cas_drift_threshold_exceeded,communication_issue,\n time_out_of_sync,disk_usage_analyzer_stuck]}], choosing the former.', 'serverTime': '2022-03-08T04:06:54.205Z'} 2022-03-08 04:09:08,782 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1646741214201, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.123.61' saw that node 'ns_1@172.23.122.119' came up. Tags: []", 'serverTime': '2022-03-08T04:06:54.201Z'} 2022-03-08 04:09:08,783 - root - ERROR - {'node': 'ns_1@172.23.123.61', 'type': 'info', 'code': 0, 'module': 'ns_cluster', 'tstamp': 1646741214010, 'shortText': 'message', 'text': "Started node add transaction by adding node 'ns_1@172.23.122.119' to nodes_wanted (group: undefined)", 'serverTime': '2022-03-08T04:06:54.010Z'} 2022-03-08 04:09:08,783 - root - ERROR - {'node': 'ns_1@172.23.122.122', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1646740948791, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.122.122' saw that node 'ns_1@172.23.122.119' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-08T04:02:28.791Z'} 2022-03-08 04:09:08,783 - root - INFO - closing all ssh connections 2022-03-08 04:09:08,784 - root - INFO - closing all memcached connections Cluster instance shutdown with force 2022-03-08 04:09:08,791 - root - INFO - ============== EventingBaseTest tearDown has completed ============== 2022-03-08 04:09:08,792 - root - INFO - will install 6.6.5-10080 on ['172.23.123.61', '172.23.122.122', '172.23.122.123', '172.23.122.119', '172.23.122.120', '172.23.122.121', '172.23.122.124'] 2022-03-08 04:09:08,795 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:09:08,797 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:09:08,798 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:09:08,801 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:09:08,803 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:09:08,805 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:09:08,808 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:09:08,905 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:09:08,906 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:09:08,913 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:09:08,913 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:09:08,919 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:09:08,921 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:09:08,922 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:09:09,262 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,269 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,271 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,277 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,280 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,287 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,293 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:09,583 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,585 - root - INFO - 172.23.122.123 *****In couchbase uninstall**** 2022-03-08 04:09:09,585 - root - INFO - CentOS 2022-03-08 04:09:09,596 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,597 - root - INFO - 172.23.122.121 *****In couchbase uninstall**** 2022-03-08 04:09:09,597 - root - INFO - CentOS 2022-03-08 04:09:09,600 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,600 - root - INFO - 172.23.122.124 *****In couchbase uninstall**** 2022-03-08 04:09:09,601 - root - INFO - CentOS 2022-03-08 04:09:09,602 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,603 - root - INFO - 172.23.123.61 *****In couchbase uninstall**** 2022-03-08 04:09:09,603 - root - INFO - CentOS 2022-03-08 04:09:09,607 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,608 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,609 - root - INFO - 172.23.122.119 *****In couchbase uninstall**** 2022-03-08 04:09:09,609 - root - INFO - CentOS 2022-03-08 04:09:09,610 - root - INFO - 172.23.122.120 *****In couchbase uninstall**** 2022-03-08 04:09:09,612 - root - INFO - CentOS 2022-03-08 04:09:09,626 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:09,627 - root - INFO - 172.23.122.122 *****In couchbase uninstall**** 2022-03-08 04:09:09,628 - root - INFO - CentOS 2022-03-08 04:09:09,731 - root - INFO - running command.raw on 172.23.122.119: killall -9 rpm 2022-03-08 04:09:09,738 - root - INFO - running command.raw on 172.23.122.122: killall -9 rpm 2022-03-08 04:09:09,747 - root - INFO - running command.raw on 172.23.122.123: killall -9 rpm 2022-03-08 04:09:09,758 - root - INFO - running command.raw on 172.23.122.121: killall -9 rpm 2022-03-08 04:09:09,768 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:09,768 - root - ERROR - rpm: no process found 2022-03-08 04:09:09,768 - root - INFO - running command.raw on 172.23.122.119: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:09,779 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:09,779 - root - ERROR - rpm: no process found 2022-03-08 04:09:09,779 - root - INFO - running command.raw on 172.23.122.122: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:09,780 - root - INFO - running command.raw on 172.23.123.61: killall -9 rpm 2022-03-08 04:09:09,786 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:09,786 - root - ERROR - rpm: no process found 2022-03-08 04:09:09,786 - root - INFO - running command.raw on 172.23.122.123: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:09,793 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:09,793 - root - ERROR - rpm: no process found 2022-03-08 04:09:09,794 - root - INFO - running command.raw on 172.23.122.121: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:09,805 - root - INFO - running command.raw on 172.23.122.124: killall -9 rpm 2022-03-08 04:09:09,815 - root - INFO - running command.raw on 172.23.122.120: killall -9 rpm 2022-03-08 04:09:09,824 - root - INFO - command executed successfully with root 2022-03-08 04:09:09,835 - root - INFO - command executed successfully with root 2022-03-08 04:09:09,839 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:09,840 - root - ERROR - rpm: no process found 2022-03-08 04:09:09,840 - root - INFO - running command.raw on 172.23.122.124: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:09,842 - root - INFO - command executed successfully with root 2022-03-08 04:09:09,848 - root - INFO - command executed successfully with root 2022-03-08 04:09:09,850 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:09,851 - root - ERROR - rpm: no process found 2022-03-08 04:09:09,851 - root - INFO - running command.raw on 172.23.122.120: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:09,898 - root - INFO - command executed successfully with root 2022-03-08 04:09:09,908 - root - INFO - command executed successfully with root 2022-03-08 04:09:09,938 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:09,939 - root - INFO - running command.raw on 172.23.122.119: rpm -e couchbase-server 2022-03-08 04:09:09,949 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:09,949 - root - INFO - running command.raw on 172.23.122.123: rpm -e couchbase-server 2022-03-08 04:09:10,001 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:10,001 - root - INFO - running command.raw on 172.23.122.121: rpm -e couchbase-server 2022-03-08 04:09:10,011 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:10,011 - root - INFO - running command.raw on 172.23.122.124: rpm -e couchbase-server 2022-03-08 04:09:10,019 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:10,019 - root - INFO - running command.raw on 172.23.122.120: rpm -e couchbase-server 2022-03-08 04:09:10,026 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:10,027 - root - INFO - running command.raw on 172.23.122.122: rpm -e couchbase-server 2022-03-08 04:09:10,060 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2022-03-08 04:09:10,060 - root - ERROR - rpm: no process found 2022-03-08 04:09:10,060 - root - INFO - running command.raw on 172.23.123.61: rm -f /var/lib/rpm/.rpm.lock 2022-03-08 04:09:10,117 - root - INFO - command executed successfully with root 2022-03-08 04:09:10,222 - root - INFO - running rpm -e to remove couchbase-server 2022-03-08 04:09:10,222 - root - INFO - running command.raw on 172.23.123.61: rpm -e couchbase-server 2022-03-08 04:09:12,951 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbas ... 2022-03-08 04:09:12,951 - root - INFO - running command.raw on 172.23.122.123: pkill -u couchbase 2022-03-08 04:09:13,004 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,004 - root - INFO - running command.raw on 172.23.122.123: ps -ef | grep couchbase 2022-03-08 04:09:13,008 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbas ... 2022-03-08 04:09:13,009 - root - INFO - running command.raw on 172.23.122.124: pkill -u couchbase 2022-03-08 04:09:13,075 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,090 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,090 - root - INFO - running command.raw on 172.23.122.124: ps -ef | grep couchbase 2022-03-08 04:09:13,092 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbas ... 2022-03-08 04:09:13,092 - root - INFO - running command.raw on 172.23.122.122: pkill -u couchbase 2022-03-08 04:09:13,095 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbas ... 2022-03-08 04:09:13,095 - root - INFO - running command.raw on 172.23.122.120: pkill -u couchbase 2022-03-08 04:09:13,100 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbas ... 2022-03-08 04:09:13,101 - root - INFO - running command.raw on 172.23.122.121: pkill -u couchbase 2022-03-08 04:09:13,121 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,121 - root - INFO - running command.raw on 172.23.122.122: ps -ef | grep couchbase 2022-03-08 04:09:13,124 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,125 - root - INFO - running command.raw on 172.23.122.121: ps -ef | grep couchbase 2022-03-08 04:09:13,161 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,162 - root - INFO - running command.raw on 172.23.122.120: ps -ef | grep couchbase 2022-03-08 04:09:13,163 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,192 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,194 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,229 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,250 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbas ... 2022-03-08 04:09:13,250 - root - INFO - running command.raw on 172.23.123.61: pkill -u couchbase 2022-03-08 04:09:13,313 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,314 - root - INFO - running command.raw on 172.23.123.61: ps -ef | grep couchbase 2022-03-08 04:09:13,388 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,855 - root - INFO - command executed with root but got an error ['warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt ... 2022-03-08 04:09:13,857 - root - INFO - running command.raw on 172.23.122.119: pkill -u couchbase 2022-03-08 04:09:13,880 - root - INFO - command executed successfully with root 2022-03-08 04:09:13,880 - root - INFO - running command.raw on 172.23.122.119: ps -ef | grep couchbase 2022-03-08 04:09:13,953 - root - INFO - command executed successfully with root 2022-03-08 04:09:14,514 - root - INFO - running command.raw on 172.23.122.123: ipcrm 2022-03-08 04:09:14,571 - root - INFO - command executed successfully with root 2022-03-08 04:09:14,579 - root - INFO - running command.raw on 172.23.122.122: ipcrm 2022-03-08 04:09:14,598 - root - INFO - running command.raw on 172.23.122.124: ipcrm 2022-03-08 04:09:14,624 - root - INFO - running command.raw on 172.23.122.120: ipcrm 2022-03-08 04:09:14,636 - root - INFO - command executed successfully with root 2022-03-08 04:09:14,637 - root - INFO - running command.raw on 172.23.122.121: ipcrm 2022-03-08 04:09:14,653 - root - INFO - command executed successfully with root 2022-03-08 04:09:14,683 - root - INFO - command executed successfully with root 2022-03-08 04:09:14,693 - root - INFO - command executed successfully with root 2022-03-08 04:09:14,854 - root - INFO - running command.raw on 172.23.123.61: ipcrm 2022-03-08 04:09:15,072 - root - INFO - command executed successfully with root thread uninstaller-thread-172.23.123.61 finished thread uninstaller-thread-172.23.122.122 finished thread uninstaller-thread-172.23.122.123 finished 2022-03-08 04:09:15,359 - root - INFO - running command.raw on 172.23.122.119: ipcrm 2022-03-08 04:09:15,414 - root - INFO - command executed successfully with root thread uninstaller-thread-172.23.122.119 finished thread uninstaller-thread-172.23.122.120 finished thread uninstaller-thread-172.23.122.121 finished thread uninstaller-thread-172.23.122.124 finished 2022-03-08 04:09:15,416 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:09:15,527 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:09:15,807 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:16,115 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:16,176 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:09:16,281 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:09:16,563 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:16,865 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:16,926 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:09:17,032 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:09:17,308 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:17,615 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:17,676 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:09:17,779 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:09:18,042 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:18,349 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:18,412 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:09:18,519 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:09:18,812 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:19,119 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:19,179 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:09:19,287 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:09:19,562 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:19,878 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:19,941 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:09:20,048 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:09:20,322 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:20,625 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:20,685 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,687 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:09:20,687 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,688 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,690 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:09:20,692 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:09:20,692 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,694 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:09:20,694 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,696 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:09:20,697 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,699 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:09:20,700 - root - INFO - ********CouchbaseServerInstaller:install 2022-03-08 04:09:20,703 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:09:20,792 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:09:20,794 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:09:20,798 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:09:20,801 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:09:20,805 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:09:20,806 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:09:20,807 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:09:21,149 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,151 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,151 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,166 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,169 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,174 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,182 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,476 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.122.123 is centos 7 *** 2022-03-08 04:09:21,479 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:09:21,487 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.122.121 is centos 7 *** 2022-03-08 04:09:21,489 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:09:21,491 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:21,493 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.122.122 is centos 7 *** --> remote server architecture_type..x86_64 *** OS version of this server 172.23.122.124 is centos 7 *** 2022-03-08 04:09:21,496 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:09:21,498 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:09:21,510 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.123.61 is centos 7 *** 2022-03-08 04:09:21,512 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:09:21,519 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.122.119 is centos 7 *** 2022-03-08 04:09:21,521 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:09:21,537 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.122.120 is centos 7 *** 2022-03-08 04:09:21,539 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:09:21,592 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:09:21,600 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:09:21,609 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:09:21,611 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:09:21,621 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:09:21,624 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:09:21,641 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:09:21,913 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,925 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,948 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,963 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,968 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,971 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:21,974 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,266 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,267 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:22,270 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,272 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:09:22,276 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,277 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:22,280 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,281 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:22,282 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,285 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,286 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:09:22,287 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:22,289 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:09:22,291 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,294 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:22,295 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:22,297 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,298 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,299 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,299 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:22,300 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,302 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:09:22,303 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,304 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,305 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm is live 2022-03-08 04:09:22,306 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:09:22,309 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:09:22,311 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:09:22,400 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:09:22,408 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:09:22,415 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:09:22,416 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:09:22,416 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:09:22,419 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:09:22,436 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:09:22,721 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,742 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,743 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,751 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,760 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,764 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:22,765 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:09:23,078 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:23,080 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,081 - root - INFO - running command.raw on 172.23.122.119: /sbin/iptables -F Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,084 - root - INFO - running command.raw on 172.23.122.124: /sbin/iptables -F 2022-03-08 04:09:23,086 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,087 - root - INFO - running command.raw on 172.23.122.121: /sbin/iptables -F 2022-03-08 04:09:23,088 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:09:23,089 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,090 - root - INFO - running command.raw on 172.23.122.120: /sbin/iptables -F Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,091 - root - INFO - running command.raw on 172.23.123.61: /sbin/iptables -F 2022-03-08 04:09:23,124 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,125 - root - INFO - running command.raw on 172.23.122.122: /sbin/iptables -F 2022-03-08 04:09:23,129 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm name : couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm product : product_version : 6.6.5-10080 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2022-03-08 04:09:23,129 - root - INFO - running command.raw on 172.23.122.123: /sbin/iptables -F 2022-03-08 04:09:23,139 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,139 - root - INFO - running command.raw on 172.23.122.119: /sbin/iptables -t nat -F 2022-03-08 04:09:23,141 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,141 - root - INFO - running command.raw on 172.23.122.124: /sbin/iptables -t nat -F 2022-03-08 04:09:23,144 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,144 - root - INFO - running command.raw on 172.23.122.120: /sbin/iptables -t nat -F 2022-03-08 04:09:23,148 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,148 - root - INFO - running command.raw on 172.23.122.121: /sbin/iptables -t nat -F 2022-03-08 04:09:23,151 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,151 - root - INFO - running command.raw on 172.23.123.61: /sbin/iptables -t nat -F 2022-03-08 04:09:23,184 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,184 - root - INFO - running command.raw on 172.23.122.122: /sbin/iptables -t nat -F 2022-03-08 04:09:23,190 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,190 - root - INFO - running command.raw on 172.23.122.123: /sbin/iptables -t nat -F 2022-03-08 04:09:23,193 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,194 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:09:23,197 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,198 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:09:23,200 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,201 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:09:23,204 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,205 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:09:23,207 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,207 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:09:23,239 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,239 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:09:23,245 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,245 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:09:23,297 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:09:23,300 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:09:23,307 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:09:23,310 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:09:23,311 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:09:23,344 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:09:23,346 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:09:23,563 - root - INFO - running command.raw on 172.23.123.61: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,570 - root - INFO - running command.raw on 172.23.122.120: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,572 - root - INFO - running command.raw on 172.23.122.124: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,574 - root - INFO - running command.raw on 172.23.122.119: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,577 - root - INFO - running command.raw on 172.23.122.121: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,618 - root - INFO - running command.raw on 172.23.122.123: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,627 - root - INFO - running command.raw on 172.23.122.122: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/10080/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm;cd /tmp;ls -lh 2022-03-08 04:09:23,631 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,631 - root - INFO - total 370M 2022-03-08 04:09:23,631 - root - INFO - -rw-r--r--. 1 root root 370M Mar 8 03:56 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:23,631 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Mar 8 02:10 hsperfdata_couchbase 2022-03-08 04:09:23,631 - root - INFO - drwxr-xr-x. 5 root root 42 Feb 25 21:12 multiple_certs_test_random74 2022-03-08 04:09:23,631 - root - INFO - drwx------. 3 root root 16 Feb 18 21:14 systemd-private-11e1d555c4a843eb8373e424e5af0247-ntpd.service-dQb74X 2022-03-08 04:09:23,635 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,636 - root - INFO - total 370M 2022-03-08 04:09:23,636 - root - INFO - drwx------. 2 root root 22 Mar 7 17:36 atop.d 2022-03-08 04:09:23,636 - root - INFO - -rw-r--r--. 1 root root 370M Mar 8 03:56 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:23,636 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Mar 7 21:20 hsperfdata_couchbase 2022-03-08 04:09:23,636 - root - INFO - drwx------. 3 root root 16 Mar 7 17:36 systemd-private-1165557e77bc43c88fb9197a0554bf35-ntpd.service-EazjL1 2022-03-08 04:09:23,638 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,638 - root - INFO - total 370M 2022-03-08 04:09:23,638 - root - INFO - -rw-r--r--. 1 root root 370M Mar 8 03:56 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:23,639 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Feb 25 22:40 hsperfdata_couchbase 2022-03-08 04:09:23,639 - root - INFO - drwx------. 3 root root 16 Feb 27 07:41 systemd-private-0ff2fb257ea049e88774c97dd009b7e8-ntpd.service-iBm9lT 2022-03-08 04:09:23,642 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,642 - root - INFO - total 370M 2022-03-08 04:09:23,642 - root - INFO - drwx------. 2 root root 22 Feb 25 23:52 atop.d 2022-03-08 04:09:23,642 - root - INFO - -rw-r--r--. 1 root root 370M Mar 8 03:56 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:23,642 - root - INFO - drwxr-xr-x. 2 root root 6 Feb 26 06:12 entbackup_172.23.122.123 2022-03-08 04:09:23,642 - root - INFO - drwxr-x---. 2 couchbase couchbase 6 Feb 26 08:06 hsperfdata_couchbase 2022-03-08 04:09:23,642 - root - INFO - drwx------. 3 root root 16 Feb 25 23:52 systemd-private-6fbf28b5c50e476fbd81534efea652e4-ntpd.service-TelTm7 2022-03-08 04:09:23,684 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,685 - root - INFO - total 370M 2022-03-08 04:09:23,685 - root - INFO - -rw-r--r--. 1 root root 370M Mar 8 03:56 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:23,685 - root - INFO - drwxr-xr-x. 2 root root 6 Feb 26 06:12 entbackup_172.23.122.123 2022-03-08 04:09:23,685 - root - INFO - drwx------. 3 root root 16 Mar 7 21:13 systemd-private-f4cb283688854909a48b80d2c02f76ff-ntpd.service-kIhAua 2022-03-08 04:09:23,692 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,692 - root - INFO - total 370M 2022-03-08 04:09:23,692 - root - INFO - drwx------. 2 root root 22 Feb 26 00:42 atop.d 2022-03-08 04:09:23,692 - root - INFO - drwxrwxrwx. 4 couchbase couchbase 4.0K Mar 5 08:58 cbbackupmgr-staging 2022-03-08 04:09:23,693 - root - INFO - -rw-r--r--. 1 root root 370M Mar 8 03:56 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:23,693 - root - INFO - drwxr-xr-x. 2 root root 6 Feb 26 06:12 entbackup_172.23.122.123 2022-03-08 04:09:23,693 - root - INFO - drwx------. 3 root root 16 Feb 26 00:42 systemd-private-3b61ebdd1e614454ac6e9b5b7af6732b-ntpd.service-goRXoX 2022-03-08 04:09:23,697 - root - INFO - *****install server *** 2022-03-08 04:09:23,697 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:23,697 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:23,701 - root - INFO - *****install server *** 2022-03-08 04:09:23,701 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:23,701 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:23,703 - root - INFO - *****install server *** 2022-03-08 04:09:23,703 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:23,703 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:23,709 - root - INFO - *****install server *** 2022-03-08 04:09:23,709 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:23,710 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:23,747 - root - INFO - *****install server *** 2022-03-08 04:09:23,748 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:23,748 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:23,756 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:23,756 - root - INFO - running command.raw on 172.23.123.61: cat /etc/redhat-release 2022-03-08 04:09:23,759 - root - INFO - *****install server *** 2022-03-08 04:09:23,759 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:23,759 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:23,760 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:23,760 - root - INFO - running command.raw on 172.23.122.120: cat /etc/redhat-release 2022-03-08 04:09:23,762 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:23,762 - root - INFO - running command.raw on 172.23.122.124: cat /etc/redhat-release 2022-03-08 04:09:23,768 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:23,769 - root - INFO - running command.raw on 172.23.122.121: cat /etc/redhat-release 2022-03-08 04:09:23,805 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:23,805 - root - INFO - running command.raw on 172.23.122.123: cat /etc/redhat-release 2022-03-08 04:09:23,811 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,812 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:09:23,817 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,817 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:09:23,818 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,819 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:09:23,820 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:23,820 - root - INFO - running command.raw on 172.23.122.122: cat /etc/redhat-release 2022-03-08 04:09:23,825 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,825 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:09:23,861 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,861 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:09:23,877 - root - INFO - command executed successfully with root 2022-03-08 04:09:23,877 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:09:24,635 - root - INFO - command executed successfully with root 2022-03-08 04:09:24,635 - root - INFO - total 370M 2022-03-08 04:09:24,636 - root - INFO - -rw-r--r--. 1 root root 370M Jan 18 16:04 couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm 2022-03-08 04:09:24,636 - root - INFO - drwx------. 3 root root 16 Feb 26 08:32 systemd-private-2bb6ab08e5504815b4fc1d8af624077a-ntpd.service-UgCL2B 2022-03-08 04:09:24,658 - root - INFO - *****install server *** 2022-03-08 04:09:24,658 - root - INFO - -->deliverable_type : rpm 2022-03-08 04:09:24,658 - root - INFO - /tmp/couchbase-server-enterprise-6.6.5-10080-centos7.x86_64.rpm or /tmp/ 2022-03-08 04:09:24,716 - root - INFO - --> execute_command: end... --> inside rpm check... 2022-03-08 04:09:24,717 - root - INFO - running command.raw on 172.23.122.119: cat /etc/redhat-release 2022-03-08 04:09:24,772 - root - INFO - command executed successfully with root 2022-03-08 04:09:24,772 - root - INFO - CentOS Linux release 7.7.1908 (Core) 2022-03-08 04:10:19,476 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:19,477 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:19,538 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:19,540 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:19,620 - root - INFO - running command.raw on 172.23.122.124: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:19,677 - root - INFO - command executed successfully with root 2022-03-08 04:10:19,678 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:19,759 - root - INFO - Starting couchbase server 2022-03-08 04:10:19,876 - root - INFO - Running systemd command on this server 2022-03-08 04:10:19,877 - root - INFO - running command.raw on 172.23.122.124: systemctl start couchbase-server.service 2022-03-08 04:10:19,905 - root - INFO - command executed successfully with root 2022-03-08 04:10:19,905 - root - INFO - 172.23.122.124:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:20,026 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:20,027 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:20,050 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:20,051 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:20,131 - root - INFO - running command.raw on 172.23.122.120: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:20,189 - root - INFO - command executed successfully with root 2022-03-08 04:10:20,190 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:20,270 - root - INFO - Starting couchbase server 2022-03-08 04:10:20,373 - root - INFO - Running systemd command on this server 2022-03-08 04:10:20,374 - root - INFO - running command.raw on 172.23.122.120: systemctl start couchbase-server.service 2022-03-08 04:10:20,398 - root - INFO - command executed successfully with root 2022-03-08 04:10:20,398 - root - INFO - 172.23.122.120:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:20,681 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:20,682 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:20,707 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:20,709 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:20,788 - root - INFO - running command.raw on 172.23.122.123: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:20,844 - root - INFO - command executed successfully with root 2022-03-08 04:10:20,844 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:20,869 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:20,869 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:20,881 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:20,882 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:20,904 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:20,905 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:20,915 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:20,915 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:20,926 - root - INFO - Starting couchbase server 2022-03-08 04:10:20,930 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:20,931 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:20,978 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:20,979 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:20,985 - root - INFO - running command.raw on 172.23.122.119: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:21,009 - root - INFO - running command.raw on 172.23.122.121: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:21,032 - root - INFO - Running systemd command on this server 2022-03-08 04:10:21,032 - root - INFO - running command.raw on 172.23.122.123: systemctl start couchbase-server.service 2022-03-08 04:10:21,042 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,043 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:21,056 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,056 - root - INFO - 172.23.122.123:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:21,056 - root - INFO - running command.raw on 172.23.122.122: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:21,064 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,065 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:21,113 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,114 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:21,123 - root - INFO - Starting couchbase server 2022-03-08 04:10:21,143 - root - INFO - Starting couchbase server 2022-03-08 04:10:21,194 - root - INFO - Starting couchbase server 2022-03-08 04:10:21,243 - root - INFO - Running systemd command on this server 2022-03-08 04:10:21,243 - root - INFO - running command.raw on 172.23.122.119: systemctl start couchbase-server.service 2022-03-08 04:10:21,250 - root - INFO - Running systemd command on this server 2022-03-08 04:10:21,250 - root - INFO - running command.raw on 172.23.122.121: systemctl start couchbase-server.service 2022-03-08 04:10:21,267 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,268 - root - INFO - 172.23.122.119:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:21,274 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,275 - root - INFO - 172.23.122.121:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:21,307 - root - INFO - Running systemd command on this server 2022-03-08 04:10:21,307 - root - INFO - running command.raw on 172.23.122.122: systemctl start couchbase-server.service 2022-03-08 04:10:21,336 - root - INFO - command executed successfully with root 2022-03-08 04:10:21,336 - root - INFO - 172.23.122.122:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:22,027 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2022-03-08 04:10:22,028 - root - INFO - Warming: RPMDB altered outside of yum 2022-03-08 04:10:22,054 - root - INFO - Checking if the directory /tmp exists or not. 2022-03-08 04:10:22,055 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2022-03-08 04:10:22,141 - root - INFO - running command.raw on 172.23.123.61: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-03-08 04:10:22,200 - root - INFO - command executed successfully with root 2022-03-08 04:10:22,200 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:22,283 - root - INFO - Starting couchbase server 2022-03-08 04:10:22,399 - root - INFO - Running systemd command on this server 2022-03-08 04:10:22,400 - root - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service 2022-03-08 04:10:22,426 - root - INFO - command executed successfully with root 2022-03-08 04:10:22,426 - root - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... 2022-03-08 04:10:24,906 - root - INFO - running command.raw on 172.23.122.124: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:24,941 - root - INFO - command executed successfully with root 2022-03-08 04:10:24,941 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:24,941 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:25,013 - root - INFO - process beam.smp is running on 172.23.122.124: with pid 101135 2022-03-08 04:10:25,070 - root - INFO - wait 30 seconds for Couchbase server to start 2022-03-08 04:10:25,401 - root - INFO - running command.raw on 172.23.122.120: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:25,426 - root - INFO - command executed successfully with root 2022-03-08 04:10:25,426 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:25,427 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:25,498 - root - INFO - process beam.smp is running on 172.23.122.120: with pid 48122 2022-03-08 04:10:25,557 - root - INFO - wait 30 seconds for Couchbase server to start 2022-03-08 04:10:26,061 - root - INFO - running command.raw on 172.23.122.123: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:26,094 - root - INFO - command executed successfully with root 2022-03-08 04:10:26,094 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:26,095 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:26,173 - root - INFO - process beam.smp is running on 172.23.122.123: with pid 31152 2022-03-08 04:10:26,229 - root - INFO - wait 30 seconds for Couchbase server to start 2022-03-08 04:10:26,273 - root - INFO - running command.raw on 172.23.122.119: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:26,280 - root - INFO - running command.raw on 172.23.122.121: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:26,307 - root - INFO - command executed successfully with root 2022-03-08 04:10:26,307 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:26,307 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:26,309 - root - INFO - command executed successfully with root 2022-03-08 04:10:26,309 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:26,309 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:26,342 - root - INFO - running command.raw on 172.23.122.122: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:26,376 - root - INFO - command executed successfully with root 2022-03-08 04:10:26,376 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:26,376 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:26,386 - root - INFO - process beam.smp is running on 172.23.122.121: with pid 107646 2022-03-08 04:10:26,391 - root - INFO - process beam.smp is running on 172.23.122.119: with pid 126910 2022-03-08 04:10:26,443 - root - INFO - wait 30 seconds for Couchbase server to start 2022-03-08 04:10:26,447 - root - INFO - wait 30 seconds for Couchbase server to start 2022-03-08 04:10:26,449 - root - INFO - process beam.smp is running on 172.23.122.122: with pid 51943 2022-03-08 04:10:26,503 - root - INFO - wait 30 seconds for Couchbase server to start 2022-03-08 04:10:27,432 - root - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-03-08 04:10:27,468 - root - INFO - command executed successfully with root 2022-03-08 04:10:27,469 - root - INFO - Couchbase server status: [] 2022-03-08 04:10:27,469 - root - INFO - Checking for process beam.smp on linux 2022-03-08 04:10:27,542 - root - INFO - process beam.smp is running on 172.23.123.61: with pid 48164 2022-03-08 04:10:27,602 - root - INFO - wait 30 seconds for Couchbase server to start thread installer-thread-172.23.123.61 finished thread installer-thread-172.23.122.122 finished thread installer-thread-172.23.122.123 finished thread installer-thread-172.23.122.119 finished thread installer-thread-172.23.122.120 finished thread installer-thread-172.23.122.121 finished thread installer-thread-172.23.122.124 finished 2022-03-08 04:10:57,636 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:10:57,638 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:10:57,640 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:10:57,643 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:10:57,645 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:10:57,648 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:10:57,649 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:10:57,747 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:10:57,748 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:10:57,750 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:10:57,755 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:10:57,756 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:10:57,757 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:10:57,759 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:10:58,080 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,083 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,095 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,098 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,111 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,114 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,117 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:10:58,420 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,428 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,431 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,434 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,447 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,453 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,459 - root - INFO - 172.23.122.122 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,459 - root - INFO - creating RestConnection(ip:172.23.122.122 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,461 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:10:58,468 - root - INFO - Done creating rest connection 2022-03-08 04:10:58,485 - root - INFO - 172.23.122.121 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,485 - root - INFO - creating RestConnection(ip:172.23.122.121 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,492 - root - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,493 - root - INFO - creating RestConnection(ip:172.23.123.61 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,495 - root - INFO - Done creating rest connection 2022-03-08 04:10:58,500 - root - INFO - 172.23.122.120 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,500 - root - INFO - creating RestConnection(ip:172.23.122.120 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,502 - root - INFO - Done creating rest connection 2022-03-08 04:10:58,508 - root - INFO - Done creating rest connection 2022-03-08 04:10:58,511 - root - INFO - 172.23.122.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,511 - root - INFO - creating RestConnection(ip:172.23.122.123 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,516 - root - INFO - 172.23.122.124 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,517 - root - INFO - creating RestConnection(ip:172.23.122.124 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,518 - root - INFO - Done creating rest connection 2022-03-08 04:10:58,525 - root - INFO - Done creating rest connection 2022-03-08 04:10:58,526 - root - INFO - 172.23.122.119 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:10:58,526 - root - INFO - creating RestConnection(ip:172.23.122.119 port:8091 ssh_username:root) and the server type is 2022-03-08 04:10:58,534 - root - INFO - Done creating rest connection 2022-03-08 04:11:01,475 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,486 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,486 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,502 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,509 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,511 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,511 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,515 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,520 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,520 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,526 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,526 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,526 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,532 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,540 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,540 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,542 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:01,546 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,546 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,554 - root - INFO - Node version in cluster 6.6.5-10080-enterprise 2022-03-08 04:11:01,555 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2022-03-08 04:11:01,719 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,720 - root - INFO - running command.raw on 172.23.122.122: rpm -qa | grep 'man-db' 2022-03-08 04:11:01,745 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,746 - root - INFO - running command.raw on 172.23.122.121: rpm -qa | grep 'man-db' 2022-03-08 04:11:01,748 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,749 - root - INFO - running command.raw on 172.23.123.61: rpm -qa | grep 'man-db' 2022-03-08 04:11:01,761 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,762 - root - INFO - running command.raw on 172.23.122.120: rpm -qa | grep 'man-db' 2022-03-08 04:11:01,771 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,772 - root - INFO - running command.raw on 172.23.122.123: rpm -qa | grep 'man-db' 2022-03-08 04:11:01,781 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,781 - root - INFO - running command.raw on 172.23.122.124: rpm -qa | grep 'man-db' 2022-03-08 04:11:01,791 - root - INFO - check if man installed on vm? 2022-03-08 04:11:01,792 - root - INFO - running command.raw on 172.23.122.119: rpm -qa | grep 'man-db' 2022-03-08 04:11:02,859 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,859 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.122.122 2022-03-08 04:11:02,860 - root - INFO - running command.raw on 172.23.122.122: unzip 2022-03-08 04:11:02,878 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,878 - root - INFO - unzip command is ready 2022-03-08 04:11:02,878 - root - INFO - Check if ntp is installed 2022-03-08 04:11:02,878 - root - INFO - This OS version centos 7 2022-03-08 04:11:02,878 - root - INFO - running command.raw on 172.23.122.122: systemctl status ntpd 2022-03-08 04:11:02,914 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,914 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.122.123 2022-03-08 04:11:02,914 - root - INFO - running command.raw on 172.23.122.123: unzip 2022-03-08 04:11:02,916 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,916 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.122.120 2022-03-08 04:11:02,917 - root - INFO - running command.raw on 172.23.122.120: unzip 2022-03-08 04:11:02,923 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,923 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.122.119 2022-03-08 04:11:02,923 - root - INFO - running command.raw on 172.23.122.119: unzip 2022-03-08 04:11:02,928 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,929 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.122.121 2022-03-08 04:11:02,929 - root - INFO - running command.raw on 172.23.122.121: unzip 2022-03-08 04:11:02,945 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,946 - root - INFO - running command.raw on 172.23.122.122: ntpstat 2022-03-08 04:11:02,954 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,955 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.122.124 2022-03-08 04:11:02,955 - root - INFO - running command.raw on 172.23.122.124: unzip 2022-03-08 04:11:02,972 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,973 - root - INFO - unzip command is ready 2022-03-08 04:11:02,973 - root - INFO - Check if ntp is installed 2022-03-08 04:11:02,974 - root - INFO - This OS version centos 7 2022-03-08 04:11:02,974 - root - INFO - running command.raw on 172.23.122.124: systemctl status ntpd 2022-03-08 04:11:02,977 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,978 - root - INFO - unzip command is ready 2022-03-08 04:11:02,978 - root - INFO - Check if ntp is installed 2022-03-08 04:11:02,978 - root - INFO - This OS version centos 7 2022-03-08 04:11:02,978 - root - INFO - running command.raw on 172.23.122.120: systemctl status ntpd 2022-03-08 04:11:02,979 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,979 - root - INFO - unzip command is ready 2022-03-08 04:11:02,980 - root - INFO - Check if ntp is installed 2022-03-08 04:11:02,980 - root - INFO - This OS version centos 7 2022-03-08 04:11:02,981 - root - INFO - running command.raw on 172.23.122.123: systemctl status ntpd 2022-03-08 04:11:02,982 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,982 - root - INFO - unzip command is ready 2022-03-08 04:11:02,982 - root - INFO - Check if ntp is installed 2022-03-08 04:11:02,983 - root - INFO - This OS version centos 7 2022-03-08 04:11:02,983 - root - INFO - running command.raw on 172.23.122.119: systemctl status ntpd 2022-03-08 04:11:02,987 - root - INFO - command executed successfully with root 2022-03-08 04:11:02,987 - root - INFO - unzip command is ready 2022-03-08 04:11:02,987 - root - INFO - Check if ntp is installed 2022-03-08 04:11:02,988 - root - INFO - This OS version centos 7 2022-03-08 04:11:02,988 - root - INFO - running command.raw on 172.23.122.121: systemctl status ntpd 2022-03-08 04:11:03,043 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,044 - root - INFO - running command.raw on 172.23.122.122: timedatectl status 2022-03-08 04:11:03,048 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,048 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,050 - root - INFO - running command.raw on 172.23.122.124: ntpstat 2022-03-08 04:11:03,051 - root - INFO - running command.raw on 172.23.122.120: ntpstat 2022-03-08 04:11:03,056 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,057 - root - INFO - running command.raw on 172.23.122.123: ntpstat 2022-03-08 04:11:03,059 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,059 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.123.61 2022-03-08 04:11:03,059 - root - INFO - running command.raw on 172.23.123.61: unzip 2022-03-08 04:11:03,061 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,061 - root - INFO - running command.raw on 172.23.122.119: ntpstat 2022-03-08 04:11:03,063 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,063 - root - INFO - running command.raw on 172.23.122.121: ntpstat 2022-03-08 04:11:03,088 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,089 - root - INFO - running command.raw on 172.23.122.122: date 2022-03-08 04:11:03,119 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,120 - root - INFO - unzip command is ready 2022-03-08 04:11:03,120 - root - INFO - Check if ntp is installed 2022-03-08 04:11:03,120 - root - INFO - This OS version centos 7 2022-03-08 04:11:03,120 - root - INFO - running command.raw on 172.23.123.61: systemctl status ntpd 2022-03-08 04:11:03,139 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,139 - root - INFO - running command.raw on 172.23.122.120: timedatectl status 2022-03-08 04:11:03,142 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,142 - root - INFO - running command.raw on 172.23.122.124: timedatectl status 2022-03-08 04:11:03,145 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,154 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,154 - root - INFO - running command.raw on 172.23.122.123: timedatectl status 2022-03-08 04:11:03,157 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,158 - root - INFO - running command.raw on 172.23.122.119: timedatectl status 2022-03-08 04:11:03,160 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,160 - root - INFO - running command.raw on 172.23.122.121: timedatectl status 2022-03-08 04:11:03,181 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,181 - root - INFO - running command.raw on 172.23.122.120: date 2022-03-08 04:11:03,188 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,188 - root - INFO - running command.raw on 172.23.123.61: ntpstat 2022-03-08 04:11:03,200 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.122.122 2022-03-08 04:11:03,227 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,228 - root - INFO - running command.raw on 172.23.122.124: date 2022-03-08 04:11:03,237 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,239 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,239 - root - INFO - running command.raw on 172.23.122.123: date 2022-03-08 04:11:03,242 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,242 - root - INFO - running command.raw on 172.23.122.119: date 2022-03-08 04:11:03,245 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,245 - root - INFO - running command.raw on 172.23.122.121: date 2022-03-08 04:11:03,261 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,283 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,284 - root - INFO - running command.raw on 172.23.123.61: timedatectl status 2022-03-08 04:11:03,288 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,292 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.122.120 2022-03-08 04:11:03,294 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,302 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,316 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.122.119 2022-03-08 04:11:03,342 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.122.124 2022-03-08 04:11:03,349 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.122.123 2022-03-08 04:11:03,357 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.122.121 2022-03-08 04:11:03,367 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,367 - root - INFO - running command.raw on 172.23.123.61: date 2022-03-08 04:11:03,423 - root - INFO - command executed successfully with root 2022-03-08 04:11:03,478 - root - INFO - ['Tue Mar 8 04:11:03 PST 2022'] IP: 172.23.123.61 thread initializer-thread-172.23.123.61 finished thread initializer-thread-172.23.122.122 finished thread initializer-thread-172.23.122.123 finished thread initializer-thread-172.23.122.119 finished thread initializer-thread-172.23.122.120 finished thread initializer-thread-172.23.122.121 finished thread initializer-thread-172.23.122.124 finished 2022-03-08 04:11:03,480 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:11:03,577 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:11:03,850 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:04,163 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:04,164 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** 2022-03-08 04:11:04,170 - root - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 04:11:04,171 - root - INFO - At least one of the node in the cluster is on pre-spock version. Not creating user since RBAC is a spock feature. 2022-03-08 04:11:04,172 - root - INFO - sleep for 5 secs. ... 2022-03-08 04:11:09,177 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2022-03-08 04:11:09,183 - root - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-08 04:11:09,185 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:11:09,285 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:11:09,562 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:09,871 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:09,934 - root - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:09,934 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:14,943 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:14,948 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:11:15,053 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:11:15,348 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:15,658 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:15,721 - root - INFO - 172.23.122.122 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:15,721 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:20,731 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:20,737 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:11:20,845 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:11:21,129 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:21,443 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:21,470 - root - INFO - 172.23.122.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:21,470 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:26,479 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:26,483 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:11:26,592 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:11:26,895 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:27,226 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:27,292 - root - INFO - 172.23.122.119 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:27,292 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:32,302 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:32,309 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:11:32,420 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:11:32,714 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:33,025 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:33,088 - root - INFO - 172.23.122.120 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:33,088 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:38,099 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:38,103 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:11:38,211 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:11:38,486 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:38,795 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:38,860 - root - INFO - 172.23.122.121 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:38,860 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:43,869 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:43,874 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:11:43,980 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:11:44,255 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:11:44,567 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-08 04:11:44,632 - root - INFO - 172.23.122.124 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-03-08 04:11:44,632 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2022-03-08 04:11:49,642 - root - INFO - -->is_ns_server_running? 2022-03-08 04:11:49,646 - root - INFO - sleep for 20 secs. sleep 20 seconds before run next test ... 2022-03-08 04:12:09,655 - root - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 2022-03-08 04:12:09,657 - root - INFO - SSH Connecting to 172.23.122.122 with username:root, attempt#1 of 5 2022-03-08 04:12:09,659 - root - INFO - SSH Connecting to 172.23.122.123 with username:root, attempt#1 of 5 2022-03-08 04:12:09,663 - root - INFO - SSH Connecting to 172.23.122.119 with username:root, attempt#1 of 5 2022-03-08 04:12:09,665 - root - INFO - SSH Connecting to 172.23.122.120 with username:root, attempt#1 of 5 2022-03-08 04:12:09,669 - root - INFO - SSH Connecting to 172.23.122.121 with username:root, attempt#1 of 5 2022-03-08 04:12:09,671 - root - INFO - SSH Connecting to 172.23.122.124 with username:root, attempt#1 of 5 2022-03-08 04:12:09,765 - root - INFO - SSH Connected to 172.23.122.122 as root 2022-03-08 04:12:09,766 - root - INFO - SSH Connected to 172.23.123.61 as root 2022-03-08 04:12:09,771 - root - INFO - SSH Connected to 172.23.122.123 as root 2022-03-08 04:12:09,771 - root - INFO - SSH Connected to 172.23.122.119 as root 2022-03-08 04:12:09,781 - root - INFO - SSH Connected to 172.23.122.120 as root 2022-03-08 04:12:09,783 - root - INFO - SSH Connected to 172.23.122.124 as root 2022-03-08 04:12:09,785 - root - INFO - SSH Connected to 172.23.122.121 as root 2022-03-08 04:12:10,109 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,124 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,131 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,143 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,145 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,148 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,159 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-08 04:12:10,464 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.122.123 2022-03-08 04:12:10,467 - root - INFO - running command.raw on 172.23.122.123: /opt/couchbase/bin/cbcollect_info 172.23.122.123-20220308-0412-diag.zip 2022-03-08 04:12:10,471 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.122.124 2022-03-08 04:12:10,472 - root - INFO - running command.raw on 172.23.122.124: /opt/couchbase/bin/cbcollect_info 172.23.122.124-20220308-0412-diag.zip 2022-03-08 04:12:10,476 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.122.122 2022-03-08 04:12:10,477 - root - INFO - running command.raw on 172.23.122.122: /opt/couchbase/bin/cbcollect_info 172.23.122.122-20220308-0412-diag.zip 2022-03-08 04:12:10,478 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.61 2022-03-08 04:12:10,481 - root - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20220308-0412-diag.zip 2022-03-08 04:12:10,496 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.122.119 2022-03-08 04:12:10,497 - root - INFO - running command.raw on 172.23.122.119: /opt/couchbase/bin/cbcollect_info 172.23.122.119-20220308-0412-diag.zip 2022-03-08 04:12:10,509 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.122.121 2022-03-08 04:12:10,510 - root - INFO - running command.raw on 172.23.122.121: /opt/couchbase/bin/cbcollect_info 172.23.122.121-20220308-0412-diag.zip 2022-03-08 04:12:10,523 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.122.120 2022-03-08 04:12:10,525 - root - INFO - running command.raw on 172.23.122.120: /opt/couchbase/bin/cbcollect_info 172.23.122.120-20220308-0412-diag.zip 2022-03-08 04:13:25,536 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.123.61 2022-03-08 04:13:25,725 - root - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20220308-0412-diag.zip 2022-03-08 04:13:25,740 - root - INFO - command executed successfully with root 2022-03-08 04:13:26,849 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.122.120 2022-03-08 04:13:27,033 - root - INFO - running command.raw on 172.23.122.120: rm -f /root/172.23.122.120-20220308-0412-diag.zip 2022-03-08 04:13:27,052 - root - INFO - command executed successfully with root 2022-03-08 04:13:27,180 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.122.121 2022-03-08 04:13:27,372 - root - INFO - running command.raw on 172.23.122.121: rm -f /root/172.23.122.121-20220308-0412-diag.zip 2022-03-08 04:13:27,389 - root - INFO - command executed successfully with root 2022-03-08 04:13:27,579 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.122.123 2022-03-08 04:13:27,774 - root - INFO - running command.raw on 172.23.122.123: rm -f /root/172.23.122.123-20220308-0412-diag.zip 2022-03-08 04:13:27,833 - root - INFO - command executed successfully with root 2022-03-08 04:13:28,655 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.122.119 2022-03-08 04:13:28,849 - root - INFO - running command.raw on 172.23.122.119: rm -f /root/172.23.122.119-20220308-0412-diag.zip 2022-03-08 04:13:28,865 - root - INFO - command executed successfully with root 2022-03-08 04:13:30,540 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.122.124 2022-03-08 04:13:30,741 - root - INFO - running command.raw on 172.23.122.124: rm -f /root/172.23.122.124-20220308-0412-diag.zip 2022-03-08 04:13:30,758 - root - INFO - command executed successfully with root 2022-03-08 04:13:43,546 - root - INFO - command executed successfully with root Downloading zipped logs from 172.23.122.122 2022-03-08 04:13:43,724 - root - INFO - running command.raw on 172.23.122.122: rm -f /root/172.23.122.122-20220308-0412-diag.zip 2022-03-08 04:13:43,780 - root - INFO - command executed successfully with root summary so far suite gsi.upgrade_gsi.UpgradeSecondaryIndex , pass 0 , fail 1 failures so far... gsi.upgrade_gsi.UpgradeSecondaryIndex.test_online_upgrade testrunner logs, diags and results are available under /data/workspace/centos-p0-2i-vset00-00-upgrade-6.6.x/logs/testrunner-22-Mar-08_03-58-01/test_1 ERROR ====================================================================== ERROR: test_online_upgrade (gsi.upgrade_gsi.UpgradeSecondaryIndex) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/gsi/upgrade_gsi.py", line 369, in test_online_upgrade rebalance.result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception File "lib/tasks/task.py", line 842, in check (status, progress) = self.rest._rebalance_status_and_progress() File "lib/membase/api/rest_client.py", line 1997, in _rebalance_status_and_progress raise RebalanceFailedException(msg) membase.api.exception.RebalanceFailedException: Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 847.795s FAILED (errors=1)