./testrunner -i /tmp/testexec.14956.ini -p gsi_type=plasma,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,GROUP=P1 -t gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_duplicate_scheduled_indexes,default_bucket=False,num_buckets=1,bucket_size=200,num_scopes=0,num_collections=0,defer_build=False,reset_services=True,GROUP=P1,use_cbbackupmgr=True Test Input params: {'default_bucket': 'False', 'num_buckets': '1', 'bucket_size': '200', 'num_scopes': '0', 'num_collections': '0', 'defer_build': 'False', 'reset_services': 'True', 'GROUP': 'P1', 'use_cbbackupmgr': 'True', 'ini': '/tmp/testexec.14956.ini', 'cluster_name': 'testexec.14956', 'spec': 'backup_and_restore', 'conf_file': 'gsi/backup_and_restore.conf', 'gsi_type': 'plasma', 'get-cbcollect-info': 'True', 'java_sdk_client': 'True', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/data/workspace/centos-p0-2i-vset00-00-collections_gsi_backup_restore_7.0_P1/logs/testrunner-21-Mar-04_22-52-10/test_1'} Run before suite setup for gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_duplicate_scheduled_indexes lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) lib/couchbase_helper/data_analysis_helper.py:821: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" pytests/security/SecretsMasterBase.py:63: DeprecationWarning: invalid escape sequence \. status, ipAddress = subprocess.getstatusoutput("ifconfig eth0 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | awk '{print $2}'") suite_setUp (gsi.backup_and_restore.BackupRestoreTests) ... -->before_suite_name:gsi.backup_and_restore.BackupRestoreTests.suite_setUp,suite: ]> 2021-03-04 22:52:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4555-enterprise 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-4555-enterprise'] 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.97, nothing to delete 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.95, nothing to delete 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.96, nothing to delete 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.94, nothing to delete 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.97:8091 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.97:8091 is running 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.95:8091 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.95:8091 is running 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.96:8091 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.96:8091 is running 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.94:8091 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.94:8091 is running 2021-03-04 22:52:12 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2021-03-04 22:52:21 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:52:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST http://172.23.123.97:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: zgrep "panic" API is accessible from localhost only/indexer.log* | wc -l 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/indexer.log*.gz: No such file or directory'] ... 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:23 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST http://172.23.123.97:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: zgrep "panic" API is accessible from localhost only/projector.log* | wc -l 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/projector.log*.gz: No such file or directory'] ... 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.123.97:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.582119537920643, 'mem_free': 3587661824, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:52:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:52:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:52:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.97, nothing to delete 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.95, nothing to delete 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.96, nothing to delete 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.94, nothing to delete 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.97:8091 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.97:8091 is running 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.95:8091 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.95:8091 is running 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.96:8091 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.96:8091 is running 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.94:8091 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.94:8091 is running 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:52:33 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:52:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl stop couchbase-server.service 2021-03-04 22:52:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:34 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:52:34 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:52:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:52:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:52:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:52:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl start couchbase-server.service 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:40 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.97:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 41549 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.97: with pid 41690 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:52:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:52:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:52:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:52:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:52:47 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:52:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl stop couchbase-server.service 2021-03-04 22:52:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:49 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:52:49 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:52:54 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:52:55 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:52:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl start couchbase-server.service 2021-03-04 22:52:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:52:55 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.95:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 41479 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.95: with pid 41667 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:53:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:53:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:01 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:01 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:53:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl stop couchbase-server.service 2021-03-04 22:53:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:03 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:53:03 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:53:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl start couchbase-server.service 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:09 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.96:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 130814 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.96: with pid 352 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:53:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:53:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl stop couchbase-server.service 2021-03-04 22:53:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:17 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:53:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:53:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl start couchbase-server.service 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:23 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.94:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 12995 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.94: with pid 13137 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:53:28 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-03-04 22:53:38 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-03-04 22:53:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:53:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:53:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:53:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:53:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:40 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:40 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:53:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl stop couchbase-server.service 2021-03-04 22:53:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:41 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:53:41 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:53:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:53:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:53:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl start couchbase-server.service 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:47 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.97:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:53:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:53:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:52 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 42220 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:53:52 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:53:52 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.97: with pid 42363 2021-03-04 22:53:52 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:53:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:53:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:53:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:53:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:53:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:53:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:53:54 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:53:54 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:53:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl stop couchbase-server.service 2021-03-04 22:53:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:53:56 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:53:56 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:54:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl start couchbase-server.service 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:02 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.95:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 42185 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.95: with pid 42326 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:54:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:54:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl stop couchbase-server.service 2021-03-04 22:54:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:10 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:54:10 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:54:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl start couchbase-server.service 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:16 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.96:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 739 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.96: with pid 921 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:54:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:54:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:54:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:54:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:23 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:54:23 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:54:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl stop couchbase-server.service 2021-03-04 22:54:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:24 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:54:24 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:54:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:54:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:54:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl start couchbase-server.service 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:30 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.94:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 13667 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.94: with pid 13809 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:54:35 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.97 port:8091 ssh_username:root, nodes/self 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '54', 'memoryTotal': 4201672704, 'memoryFree': 3646013440, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 22:54:46 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.123.97 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1891 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.123.97,8091,['n1ql', 'kv', 'index']) 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.123.97: 8091:hostname=172.23.123.97&user=Administrator&password=password&services=n1ql%2Ckv%2Cindex 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.97:8091:port=8091&username=Administrator&password=password 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:54:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.95 port:8091 ssh_username:root, nodes/self 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '39', 'memoryTotal': 4201672704, 'memoryFree': 3642322944, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.95:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.95', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 22:54:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.95:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.95:8091:port=8091&username=Administrator&password=password 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:54:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.95:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.95:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.96 port:8091 ssh_username:root, nodes/self 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '29', 'memoryTotal': 4201672704, 'memoryFree': 3524648960, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.96:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.96', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 22:54:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.96:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.96:8091:port=8091&username=Administrator&password=password 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:54:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.96:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.96:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.94 port:8091 ssh_username:root, nodes/self 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '14', 'memoryTotal': 4201672704, 'memoryFree': 3629731840, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.94:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.94', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 22:54:49 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.94:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.94:8091:port=8091&username=Administrator&password=password 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:54:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.94:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.94:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 22:54:50 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 22:54:50 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.123.97 **** 2021-03-04 22:54:50 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.123.97:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-04 22:54:50 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-03-04 22:54:50 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:54:55 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-03-04 22:54:55 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-03-04 22:54:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:54:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:54:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.123.97 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:54:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:54:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:54:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:54:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:54:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:54:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:54:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:54:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.123.97:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.91530123519032, 'mem_free': 3582771200, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.123.97 cluster is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.123.97 is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.123.95 cluster is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.123.95 is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.123.96 cluster is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.123.96 is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 172.23.123.94 cluster is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 172.23.123.94 is healthy 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-03-04 22:55:04 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:55:09 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-03-04 22:55:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:55:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:55:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:55:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:55:26 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2021-03-04 22:55:26 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:55:26 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:172.23.123.97 port:8091 ssh_username:root 2021-03-04 22:55:26 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2021-03-04 22:55:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:55:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:55:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:55:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /tmp/backup/ 2021-03-04 22:55:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:55:58 | INFO | MainProcess | MainThread | [rest_client.create_bucket] http://172.23.123.97:8091/pools/default/buckets with param: name=test_bucket_1&ramQuotaMB=200&authType=none&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-04 22:55:58 | INFO | MainProcess | MainThread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket_1 2021-03-04 22:55:58 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX primary_idx_1 ON default:test_bucket_1 USING GSI 2021-03-04 22:55:58 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+primary_idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-04 22:56:06 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.237395416s 2021-03-04 22:56:09 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1_1_1 ON default:test_bucket_1(firstName,lastName) USING GSI WITH {'defer_build': False} 2021-03-04 22:56:09 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1_1_1+ON+default%3Atest_bucket_1%28firstName%2ClastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.646582572s 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [backup_and_restore.setUp] Enabling debug logs for Indexer 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.settings.log_level': 'debug'} set 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.log_level': 'debug'} set 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:56:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.123.97:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 10.70794214666328, 'mem_free': 3423199232, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:56:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:56:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:56:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:56:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:56:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:56:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:22 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-03-04 22:56:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket_1'] on 172.23.123.97 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.123.97 existing buckets : [] 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket_1 from 172.23.123.97 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.95, nothing to delete 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.96, nothing to delete 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.94, nothing to delete 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.97:8091 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.97:8091 is running 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.95:8091 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.95:8091 is running 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.96:8091 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.96:8091 is running 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.94:8091 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.94:8091 is running 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:56:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:56:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:56:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:56:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:25 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:56:25 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:56:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl stop couchbase-server.service 2021-03-04 22:56:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:27 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:56:27 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:56:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:56:33 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:56:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl start couchbase-server.service 2021-03-04 22:56:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:33 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.97:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 46523 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.97: with pid 46667 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:56:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:56:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:39 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:56:39 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:56:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl stop couchbase-server.service 2021-03-04 22:56:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:41 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:56:41 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:56:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl start couchbase-server.service 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:47 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.95:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 46087 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.95: with pid 46229 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:56:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:56:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:56:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:56:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:56:54 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:56:54 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:56:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl stop couchbase-server.service 2021-03-04 22:56:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:56:55 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:56:55 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:57:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:57:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:57:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl start couchbase-server.service 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:01 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.96:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:57:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 4445 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.96: with pid 4590 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:57:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:57:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:08 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:57:08 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:57:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl stop couchbase-server.service 2021-03-04 22:57:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:10 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:57:10 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:57:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:57:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:57:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:57:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl start couchbase-server.service 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:16 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.123.94:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 17509 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.94: with pid 17653 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-03-04 22:57:21 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... ./lib/log_scanner.py:14: DeprecationWarning: invalid escape sequence \s "Basic\s[a-zA-Z]\{10,\}==", ./lib/log_scanner.py:15: DeprecationWarning: invalid escape sequence \[ "Menelaus-Auth-User:\[", ok ---------------------------------------------------------------------- Ran 1 test in 320.764s OK test_backup_restore_with_duplicate_scheduled_indexes (gsi.backup_and_restore.BackupRestoreTests) ... Cluster instance shutdown with force -->result: 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4555-enterprise 2021-03-04 22:57:32 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_backup_restore_with_duplicate_scheduled_indexes============== 2021-03-04 22:57:32 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.97, nothing to delete 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.95, nothing to delete 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.96, nothing to delete 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.94, nothing to delete 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.97:8091 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.97:8091 is running 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.95:8091 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.95:8091 is running 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.96:8091 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.96:8091 is running 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.94:8091 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.94:8091 is running 2021-03-04 22:57:32 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 2021-03-04 22:57:41 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:57:41 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:57:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 2021-03-04 22:57:41 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:57:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-04 22:57:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-04 22:57:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:57:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:57:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:57:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:57:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:57:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:57:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:57:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:57:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_backup_restore_with_duplicate_scheduled_indexes ============== 2021-03-04 22:57:50 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.97, nothing to delete 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.95, nothing to delete 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.96, nothing to delete 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.94, nothing to delete 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.97:8091 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.97:8091 is running 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.95:8091 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.95:8091 is running 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.96:8091 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.96:8091 is running 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.94:8091 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.94:8091 is running 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_backup_restore_with_duplicate_scheduled_indexes ============== 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:57:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:57:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:57:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:52 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:57:52 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:57:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl stop couchbase-server.service 2021-03-04 22:57:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:53 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:57:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:57:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:57:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:57:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl start couchbase-server.service 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:57:59 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.97:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:58:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:58:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:04 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 48739 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:58:04 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.97: with pid 48913 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:58:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:58:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:06 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:06 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:58:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl stop couchbase-server.service 2021-03-04 22:58:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:08 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:58:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:58:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl start couchbase-server.service 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:14 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.95:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 48379 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.95: with pid 48520 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:58:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:58:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:58:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:58:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:21 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:21 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:58:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl stop couchbase-server.service 2021-03-04 22:58:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:22 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:58:22 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:58:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:58:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:58:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl start couchbase-server.service 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:28 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.96:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:58:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:58:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:33 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 6557 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:58:33 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.96: with pid 6731 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:58:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:58:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:35 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:35 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:58:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl stop couchbase-server.service 2021-03-04 22:58:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:37 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:58:37 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:42 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:58:43 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:58:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl start couchbase-server.service 2021-03-04 22:58:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:43 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.94:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 19802 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.94: with pid 19944 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:58:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:58:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:58:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:58:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:58:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:58:59 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:58:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl stop couchbase-server.service 2021-03-04 22:59:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:01 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:59:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 22:59:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl start couchbase-server.service 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:07 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.97:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 49526 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.97: with pid 49668 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:59:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:59:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:59:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:59:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:14 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:14 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:59:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl stop couchbase-server.service 2021-03-04 22:59:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:15 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:59:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:59:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:59:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:59:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl start couchbase-server.service 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:21 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.95:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:59:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:59:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:26 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 49132 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:59:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.95: with pid 49274 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:59:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:28 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:59:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl stop couchbase-server.service 2021-03-04 22:59:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:30 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:59:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 22:59:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl start couchbase-server.service 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:36 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.96:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 7347 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.96: with pid 7489 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:59:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 22:59:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl stop couchbase-server.service 2021-03-04 22:59:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:44 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 22:59:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 22:59:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl start couchbase-server.service 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:50 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.94:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 20464 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.94: with pid 20606 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 22:59:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.97 port:8091 ssh_username:root, nodes/self 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '54', 'memoryTotal': 4201672704, 'memoryFree': 3633778688, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 23:00:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.123.97 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1891 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.123.97,8091,['n1ql', 'kv', 'index']) 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.123.97: 8091:hostname=172.23.123.97&user=Administrator&password=password&services=n1ql%2Ckv%2Cindex 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.97:8091:port=8091&username=Administrator&password=password 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:00:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.95 port:8091 ssh_username:root, nodes/self 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '39', 'memoryTotal': 4201672704, 'memoryFree': 3635494912, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.95:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.95', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 23:00:07 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.95:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.95:8091:port=8091&username=Administrator&password=password 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.95:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.95:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.96 port:8091 ssh_username:root, nodes/self 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '29', 'memoryTotal': 4201672704, 'memoryFree': 3530461184, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.96:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.96', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 23:00:08 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.96:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.96:8091:port=8091&username=Administrator&password=password 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.96:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.96:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.123.94 port:8091 ssh_username:root, nodes/self 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '14', 'memoryTotal': 4201672704, 'memoryFree': 3640868864, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.94:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4555-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.123.94', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-03-04 23:00:08 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.123.94:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-04 23:00:08 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.123.94:8091:port=8091&username=Administrator&password=password 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.94:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.94:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-04 23:00:09 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-04 23:00:09 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.123.97 **** 2021-03-04 23:00:09 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.123.97:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-04 23:00:09 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-03-04 23:00:09 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 23:00:14 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-03-04 23:00:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-03-04 23:00:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:00:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.123.97 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_backup_restore_with_duplicate_scheduled_indexes ============== 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:00:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:00:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:00:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:00:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.123.97:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.675062972292191, 'mem_free': 3574464512, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.123.97 cluster is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.123.97 is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.123.95 cluster is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.123.95 is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.123.96 cluster is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.123.96 is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.123.94 cluster is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.123.94 is healthy 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-03-04 23:00:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 23:00:29 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-03-04 23:00:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:00:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:00:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:00:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:00:48 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-03-04 23:00:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 23:00:48 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:172.23.123.97 port:8091 ssh_username:root 2021-03-04 23:00:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-03-04 23:01:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:01:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:01:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:01:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:01:19 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 23:01:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:01:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:01:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /tmp/backup/ 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [rest_client.create_bucket] http://172.23.123.97:8091/pools/default/buckets with param: name=test_bucket_1&ramQuotaMB=200&authType=none&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket_1 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX primary_idx_1 ON default:test_bucket_1 USING GSI 2021-03-04 23:01:20 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+primary_idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-04 23:01:28 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.95367631s 2021-03-04 23:01:29 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1_1_1 ON default:test_bucket_1(firstName,lastName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:29 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1_1_1+ON+default%3Atest_bucket_1%28firstName%2ClastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.691644161s 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [backup_and_restore.setUp] Enabling debug logs for Indexer 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.settings.log_level': 'debug'} set 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.log_level': 'debug'} set 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_1_1_1 ON default:test_bucket_1 USING GSI 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_1_1_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 24.43344ms 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX primary_idx_1 ON default:test_bucket_1 USING GSI 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+primary_idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 40.545184ms 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [backup_and_restore._drop_indexes] 'status' 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.debug.enableBackgroundIndexCreation': False} set 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:01:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:01:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_0 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_1 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_2 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_2 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_0+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_4 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_4 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_3 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_1 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_5 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_5 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_4 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_4+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_7 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_7 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_2 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_2+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_6 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_6 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_5 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_5+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_3+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_9 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_9 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_8 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_8 ON default:test_bucket_1(firstName) USING GSI WITH {'defer_build': False} 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_6 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_6+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_7 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_7+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_9 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_9+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | INFO | MainProcess | ThreadPoolExecutor-0_8 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_8+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_1 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_1+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "89050c2f-85d4-46f0-a040-71791f2b8329",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_1, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "157.04499ms","executionTime": "156.904876ms","resultCount": 0,"resultSize": 0,"serviceLoad": 62,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_5 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_5+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "c186718f-6d52-4162-b060-7534888cf70f",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Encountered transient error. Index creation will be retried in background. Error: Index idx_5 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket_1.\\n"}],\n"status": "errors",\n"metrics": {"elapsedTime": "150.008523ms","executionTime": "149.968396ms","resultCount": 0,"resultSize": 0,"serviceLoad": 56,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_4 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_4+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "f78fd584-a85c-4d96-9314-d2fcb6eb5464",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_4, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "161.562507ms","executionTime": "161.521328ms","resultCount": 0,"resultSize": 0,"serviceLoad": 50,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_2 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_2+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "a85e93d8-3809-4b1d-8487-3286f091963a",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_2, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "184.644228ms","executionTime": "184.580763ms","resultCount": 0,"resultSize": 0,"serviceLoad": 43,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_3 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_3+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "edcc2b24-035a-40ba-a1ff-6a76dbf936e8",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_3, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "269.64121ms","executionTime": "269.58135ms","resultCount": 0,"resultSize": 0,"serviceLoad": 31,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_8 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_8+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "975e0e54-32d2-49ce-b4c2-5f6d8c267f17",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_8, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "256.219597ms","executionTime": "256.15398ms","resultCount": 0,"resultSize": 0,"serviceLoad": 37,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_7 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_7+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "919b86b9-73c7-4f20-ad63-6b690c33dc38",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_7, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "275.230083ms","executionTime": "275.149216ms","resultCount": 0,"resultSize": 0,"serviceLoad": 25,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_6 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_6+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "222ba096-f1d5-470d-8123-f74c98e34990",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_6, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "279.146931ms","executionTime": "279.096106ms","resultCount": 0,"resultSize": 0,"serviceLoad": 18,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:33 | ERROR | MainProcess | ThreadPoolExecutor-0_9 | [rest_client._http_request] POST http://172.23.123.97:8093/query?statement=CREATE+INDEX+idx_9+ON+default%3Atest_bucket_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "455e755d-890f-45ee-8148-99b53b01aab0",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_9, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "271.613245ms","executionTime": "271.546251ms","resultCount": 0,"resultSize": 0,"serviceLoad": 12,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-04 23:01:34 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.737221991s 2021-03-04 23:01:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 60 secs. Wait till indexes available in stats ... 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_duplicate_scheduled_indexes] getIndexStatus result: {'code': 'success', 'status': [{'defnId': 10196978413897706231, 'instId': 9111348947353448071, 'name': 'idx_0', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket_1`(`firstName`)', 'hosts': ['127.0.0.1:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'127.0.0.1:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 15882176148337759904, 'instId': 10356839196362380692, 'name': 'idx_5', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Error', 'definition': 'CREATE INDEX `idx_5` ON `test_bucket_1`(`firstName`)', 'hosts': ['127.0.0.1:8091'], 'error': 'Index idx_5 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket_1.', 'completion': 0, 'progress': 0, 'scheduled': True, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'127.0.0.1:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_5', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}]} 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_duplicate_scheduled_indexes] Index status: [{'defnId': 10196978413897706231, 'instId': 9111348947353448071, 'name': 'idx_0', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket_1`(`firstName`)', 'hosts': ['127.0.0.1:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'127.0.0.1:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 15882176148337759904, 'instId': 10356839196362380692, 'name': 'idx_5', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Error', 'definition': 'CREATE INDEX `idx_5` ON `test_bucket_1`(`firstName`)', 'hosts': ['127.0.0.1:8091'], 'error': 'Index idx_5 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket_1.', 'completion': 0, 'progress': 0, 'scheduled': True, 'partitioned': False, 'numPartition': 1, 'partitionMap': {'127.0.0.1:8091': [0]}, 'numReplica': 0, 'indexName': 'idx_5', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}] 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select * from system:indexes 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+%2A+from+system%3Aindexes 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.893416ms 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_duplicate_scheduled_indexes] Query result: [{'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '3212b7adef34ec43', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_9', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '3087d47bc92aaf5a', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_2', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '6276a5cad5b54273', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_4', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '8d82f1d76eb2eaf7', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_0', 'namespace_id': 'default', 'state': 'online', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'dc68d30f3dda96a0', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'message': 'Index idx_5 will retry building in the background for reason: Build Already In Progress. Keyspace test_bucket_1.\n', 'name': 'idx_5', 'namespace_id': 'default', 'state': 'pending', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '88c862c2fcb8f260', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_3', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '3957457e3cd12cab', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_6', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '7b3f7ed340c39869', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_7', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': '4b881f7db011159d', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_1', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}, {'indexes': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'e0848a9661b2d228', 'index_key': ['`firstName`'], 'keyspace_id': 'test_bucket_1', 'name': 'idx_8', 'namespace_id': 'default', 'state': 'scheduled for creation', 'using': 'gsi'}}] 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:02:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:02:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.123.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.123.97 port:8091 ssh_username:root] 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.123.97:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.813074565883555, 'mem_free': 3011067904, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:02:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:02:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:02:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:02:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:02:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_backup_restore_with_duplicate_scheduled_indexes ============== 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket_1'] on 172.23.123.97 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.123.97 existing buckets : [] 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket_1 from 172.23.123.97 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.95, nothing to delete 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.96, nothing to delete 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.123.94, nothing to delete 2021-03-04 23:02:46 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.97:8091 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.97:8091 is running 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.95:8091 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.95:8091 is running 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.96:8091 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.96:8091 is running 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.123.94:8091 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.123.94:8091 is running 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_backup_restore_with_duplicate_scheduled_indexes ============== 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:02:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:02:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:02:48 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 23:02:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl stop couchbase-server.service 2021-03-04 23:02:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:02:50 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 23:02:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:02:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.97 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl start couchbase-server.service 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:02:56 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.97:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 53952 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.97: with pid 54138 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:03:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:03:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:03:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:03:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:03 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:03:03 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 23:03:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl stop couchbase-server.service 2021-03-04 23:03:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:04 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 23:03:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 23:03:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 23:03:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 23:03:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:03:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:03:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:10 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.95 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:03:10 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:10 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 23:03:11 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 23:03:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl start couchbase-server.service 2021-03-04 23:03:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:11 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.95:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 53465 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.95: with pid 53607 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:03:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:03:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:17 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:03:17 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 23:03:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl stop couchbase-server.service 2021-03-04 23:03:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:19 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 23:03:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:03:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.96 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl start couchbase-server.service 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:25 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.96:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 11445 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.96: with pid 11587 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:03:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:03:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:03:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:31 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:03:32 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-03-04 23:03:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl stop couchbase-server.service 2021-03-04 23:03:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:33 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-03-04 23:03:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-04 23:03:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-03-04 23:03:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-03-04 23:03:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:03:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.123.94 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl start couchbase-server.service 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:39 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.123.94:sleep for 5 secs. waiting for couchbase server to come up ... 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24702 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.123.94: with pid 24845 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-03-04 23:03:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-03-04 23:03:54 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.97 with username:root, attempt#1 of 5 2021-03-04 23:03:54 | INFO | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.95 with username:root, attempt#1 of 5 2021-03-04 23:03:54 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.96 with username:root, attempt#1 of 5 2021-03-04 23:03:54 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.123.94 with username:root, attempt#1 of 5 2021-03-04 23:03:54 | INFO | MainProcess | Thread-370 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.97 as root 2021-03-04 23:03:54 | INFO | MainProcess | Thread-372 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.96 as root 2021-03-04 23:03:54 | INFO | MainProcess | Thread-371 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.95 as root 2021-03-04 23:03:54 | INFO | MainProcess | Thread-373 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.123.94 as root 2021-03-04 23:03:55 | INFO | MainProcess | Thread-373 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:55 | INFO | MainProcess | Thread-372 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:55 | INFO | MainProcess | Thread-371 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:55 | INFO | MainProcess | Thread-370 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-04 23:03:55 | INFO | MainProcess | Thread-373 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.94 2021-03-04 23:03:55 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: /opt/couchbase/bin/cbcollect_info 172.23.123.94-20210304-2303-diag.zip 2021-03-04 23:03:55 | INFO | MainProcess | Thread-372 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.96 2021-03-04 23:03:55 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: /opt/couchbase/bin/cbcollect_info 172.23.123.96-20210304-2303-diag.zip 2021-03-04 23:03:55 | INFO | MainProcess | Thread-371 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.95 2021-03-04 23:03:55 | INFO | MainProcess | Thread-371 | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: /opt/couchbase/bin/cbcollect_info 172.23.123.95-20210304-2303-diag.zip 2021-03-04 23:03:55 | INFO | MainProcess | Thread-370 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.123.97 2021-03-04 23:03:55 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: /opt/couchbase/bin/cbcollect_info 172.23.123.97-20210304-2303-diag.zip 2021-03-04 23:05:20 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.123.94 2021-03-04 23:05:21 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] running command.raw on 172.23.123.94: rm -f /root/172.23.123.94-20210304-2303-diag.zip 2021-03-04 23:05:21 | INFO | MainProcess | Thread-373 | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:05:23 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.123.96 2021-03-04 23:05:24 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] running command.raw on 172.23.123.96: rm -f /root/172.23.123.96-20210304-2303-diag.zip 2021-03-04 23:05:24 | INFO | MainProcess | Thread-372 | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:05:25 | INFO | MainProcess | Thread-371 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.123.95 2021-03-04 23:05:25 | INFO | MainProcess | Thread-371 | [remote_util.execute_command_raw] running command.raw on 172.23.123.95: rm -f /root/172.23.123.95-20210304-2303-diag.zip 2021-03-04 23:05:25 | INFO | MainProcess | Thread-371 | [remote_util.execute_command_raw] command executed successfully with root 2021-03-04 23:05:28 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.123.97 2021-03-04 23:05:29 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] running command.raw on 172.23.123.97: rm -f /root/172.23.123.97-20210304-2303-diag.zip 2021-03-04 23:05:29 | INFO | MainProcess | Thread-370 | [remote_util.execute_command_raw] command executed successfully with root summary so far suite gsi.backup_and_restore.BackupRestoreTests , pass 0 , fail 1 failures so far... gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_duplicate_scheduled_indexes testrunner logs, diags and results are available under /data/workspace/centos-p0-2i-vset00-00-collections_gsi_backup_restore_7.0_P1/logs/testrunner-21-Mar-04_22-52-10/test_1 FAIL ====================================================================== FAIL: test_backup_restore_with_duplicate_scheduled_indexes (gsi.backup_and_restore.BackupRestoreTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/gsi/backup_and_restore.py", line 1674, in test_backup_restore_with_duplicate_scheduled_indexes self.assertTrue(idx in index_names, f"Index {idx} not available in {index_names} ") AssertionError: False is not true : Index idx_1 not available in {'idx_0', 'idx_5'} ---------------------------------------------------------------------- Ran 1 test in 382.777s