/Users/lfc/venv3_with_cb3/bin/python "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 60378 --file /Users/lfc/testrunner/testrunner.py -i b/resources/tmp.ini -p java_sdk_client=True -t gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_scheduled_partition_indexes,default_bucket=False,num_buckets=1,bucket_size=200,num_scopes=0,num_collections=0,defer_build=False,reset_services=False,GROUP=P0,use_cbbackupmgr=True,nodes_init=2,services_init=kv:index:n1ql-index:n1ql Connected to pydev debugger (build 203.7148.72) INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_scheduled_partition_indexes,default_bucket=False,num_buckets=1,bucket_size=200,num_scopes=0,num_collections=0,defer_build=False,reset_services=False,GROUP=P0,use_cbbackupmgr=True,nodes_init=2,services_init=kv:index:n1ql-index:n1ql INFO:__main__:Logs folder: /Users/lfc/testrunner/logs/testrunner-21-Mar-10_11-55-49/test_1 {'cluster_name': 'tmp', 'ini': 'b/resources/tmp.ini', 'java_sdk_client': 'True', 'num_nodes': 3} Logs will be stored at /Users/lfc/testrunner/logs/testrunner-21-Mar-10_11-55-49/test_1 ./testrunner -i b/resources/tmp.ini -p java_sdk_client=True -t gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_scheduled_partition_indexes,default_bucket=False,num_buckets=1,bucket_size=200,num_scopes=0,num_collections=0,defer_build=False,reset_services=False,GROUP=P0,use_cbbackupmgr=True,nodes_init=2,services_init=kv:index:n1ql-index:n1ql Test Input params: {'default_bucket': 'False', 'num_buckets': '1', 'bucket_size': '200', 'num_scopes': '0', 'num_collections': '0', 'defer_build': 'False', 'reset_services': 'False', 'GROUP': 'P0', 'use_cbbackupmgr': 'True', 'nodes_init': '2', 'services_init': 'kv:index:n1ql-index:n1ql', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'java_sdk_client': 'True', 'num_nodes': 3, 'case_number': 1, 'logs_folder': '/Users/lfc/testrunner/logs/testrunner-21-Mar-10_11-55-49/test_1'} Run before suite setup for gsi.backup_and_restore.BackupRestoreTests.test_backup_restore_with_scheduled_partition_indexes -->before_suite_name:gsi.backup_and_restore.BackupRestoreTests.suite_setUp,suite: ]> 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 suite_setUp (gsi.backup_and_restore.BackupRestoreTests) ... 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4614-enterprise 2021-03-10 11:55:50 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-03-10 11:55:50 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.102, nothing to delete 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.103, nothing to delete 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.102:8091 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.102:8091 is running 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.103:8091 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.103:8091 is running 2021-03-10 11:55:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] Building docker image with java sdk client 2021-03-10 11:55:56 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:55:56 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 2021-03-10 11:55:56 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:55:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:55:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:55:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:55:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:55:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:55:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-03-10 11:56:01 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.102, nothing to delete 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.103, nothing to delete 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.102:8091 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.102:8091 is running 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.103:8091 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.103:8091 is running 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-03-10 11:56:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.101 port:8091 ssh_username:root, nodes/self 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '15', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 2825, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '10.112.190.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4614-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 980, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.190.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 2825, 'curr_items': 0} 2021-03-10 11:56:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.190.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.190.101 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1636 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.190.101,8091,['kv', 'index', 'n1ql']) 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.190.101: 8091:hostname=10.112.190.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.101:8091:port=8091&username=Administrator&password=password 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:56:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.102 port:8091 ssh_username:root, nodes/self 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '21', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 2825, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '10.112.190.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4614-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 980, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.190.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 2825, 'curr_items': 0} 2021-03-10 11:56:03 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.190.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1892 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-10 11:56:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.102:8091:port=8091&username=Administrator&password=password 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.103 port:8091 ssh_username:root, nodes/self 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '20', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 2825, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '10.112.190.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4614-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 980, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.190.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 2825, 'curr_items': 0} 2021-03-10 11:56:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.190.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1892 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.103:8091:port=8091&username=Administrator&password=password 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-10 11:56:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:56:05 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-10 11:56:05 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.190.101 **** 2021-03-10 11:56:05 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://10.112.190.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-10 11:56:05 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-03-10 11:56:05 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-10 11:56:10 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-03-10 11:56:10 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-03-10 11:56:11 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.190.102:8091 to cluster 2021-03-10 11:56:11 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.190.102:8091 to this cluster @10.112.190.101:8091 2021-03-10 11:56:21 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-03-10 11:56:21 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-10 11:56:35 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.190.101,ns_1@10.112.190.102', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-03-10 11:56:35 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-03-10 11:56:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-10 11:56:35 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-03-10 11:56:45 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-03-10 11:56:45 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.037219047546387 sec 2021-03-10 11:56:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:56:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:56:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.190.101 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:56:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:56:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.39832285115304, 'mem_free': 2419523584, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.190.102:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.321138211382114, 'mem_free': 2534203392, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.190.101 cluster is healthy 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.190.101 is healthy 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.190.102 cluster is healthy 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.190.102 is healthy 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 10.112.190.103 cluster is healthy 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 10.112.190.103 is healthy 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-03-10 11:56:51 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-10 11:56:56 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-03-10 11:56:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:56:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:56:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:56:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:04 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED 2021-03-10 11:57:04 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:57:04 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:10.112.190.101 port:8091 ssh_username:root 2021-03-10 11:57:04 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ... 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:57:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: rm -rf /tmp/backup/ 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.102: rm -rf /tmp/backup/ 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [rest_client.create_bucket] http://10.112.190.101:8091/pools/default/buckets with param: name=test_bucket_1&ramQuotaMB=200&authType=none&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [rest_client.create_bucket] 0.03 seconds to create bucket test_bucket_1 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX primary_idx_1 ON default:test_bucket_1 USING GSI 2021-03-10 11:57:35 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+primary_idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 11:57:40 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.346148769s 2021-03-10 11:57:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1_1_1 ON default:test_bucket_1(firstName,lastName) USING GSI WITH {'defer_build': False} 2021-03-10 11:57:44 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1_1_1+ON+default%3Atest_bucket_1%28firstName%2ClastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.158364486s 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [backup_and_restore.setUp] Enabling debug logs for Indexer 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.settings.log_level': 'debug'} set 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.log_level': 'debug'} set 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-10 11:57:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.849948612538541, 'mem_free': 2403414016, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 10.112.190.102:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 5.888538380651945, 'mem_free': 2411450368, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:57:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:57:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:57:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:57:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:57:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:57:52 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-03-10 11:57:52 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['test_bucket_1'] on 10.112.190.101 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 10.112.190.101 existing buckets : [] 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : test_bucket_1 from 10.112.190.101 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.102, nothing to delete 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.103, nothing to delete 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.190.101,ns_1@10.112.190.102', 'ejectedNodes': 'ns_1@10.112.190.102', 'user': 'Administrator', 'password': 'password'} 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-03-10 11:57:53 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-03-10 11:58:03 | INFO | MainProcess | MainThread | [rest_client.print_UI_logs] Latest logs from UI on 10.112.190.101: 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615357681114, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {worker_died,\n {\'EXIT\',<0.8255.0>,\n {rebalance_failed,\n {service_error,\n <<"Bucket does not exist or temporarily unavailable for creating new index. Please retry the operation at a later time (err=Bucket Not Found).">>}}}}}.\nRebalance Operation Id = 861b4e21d210c5c7325674d606c366ba', 'serverTime': '2021-03-09T22:28:01.114Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615357673827, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@10.112.190.101'], EjectNodes = ['ns_1@10.112.190.102'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 861b4e21d210c5c7325674d606c366ba", 'serverTime': '2021-03-09T22:27:53.827Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1615357673824, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@10.112.190.101\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\nStarting rebalance from test, ejected nodes [\'ns_1@10.112.190.102\']', 'serverTime': '2021-03-09T22:27:53.824Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 11, 'module': 'menelaus_web', 'tstamp': 1615357673791, 'shortText': 'message', 'text': 'Deleted bucket "test_bucket_1"\n', 'serverTime': '2021-03-09T22:27:53.791Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1615357672491, 'shortText': 'message', 'text': 'Shutting down bucket "test_bucket_1" on \'ns_1@10.112.190.101\' for deletion', 'serverTime': '2021-03-09T22:27:52.491Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1615357656214, 'shortText': 'message', 'text': 'Bucket "test_bucket_1" loaded on node \'ns_1@10.112.190.101\' in 0 seconds.', 'serverTime': '2021-03-09T22:27:36.214Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 12, 'module': 'menelaus_web', 'tstamp': 1615357655633, 'shortText': 'message', 'text': 'Created bucket "test_bucket_1" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,209715200},\n {durability_min_level,none},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', 'serverTime': '2021-03-09T22:27:35.633Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 0, 'module': 'mb_master', 'tstamp': 1615357606795, 'shortText': 'message', 'text': "I'm the only node, so I'm the master. (repeated 1 times, last seen 14.30105 secs ago)", 'serverTime': '2021-03-09T22:26:46.795Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1615357606795, 'shortText': 'message', 'text': 'Enabled auto-failover with timeout 120 and max count 1 (repeated 1 times, last seen 14.297629 secs ago)', 'serverTime': '2021-03-09T22:26:46.795Z'} 2021-03-10 11:58:03 | ERROR | MainProcess | MainThread | [rest_client.print_UI_logs] {'node': 'ns_1@10.112.190.101', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1615357606099, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@10.112.190.101\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\n2021-03-10 11:56:46.056594 : suite_setUp started ', 'serverTime': '2021-03-09T22:26:46.099Z'} 2021-03-10 11:58:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-03-10 11:58:03 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force ok ---------------------------------------------------------------------- Ran 1 test in 134.057s OK test_backup_restore_with_scheduled_partition_indexes (gsi.backup_and_restore.BackupRestoreTests) ... -->result: 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4614-enterprise 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-4614-enterprise', '7.0.0-4614-enterprise'] 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_backup_restore_with_scheduled_partition_indexes============== 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.102, nothing to delete 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.103, nothing to delete 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.190.101,ns_1@10.112.190.102', 'ejectedNodes': 'ns_1@10.112.190.102', 'user': 'Administrator', 'password': 'password'} 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2021-03-10 11:58:04 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-10 11:58:24 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 20.02 seconds 2021-03-10 11:58:24 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:10.112.190.101 port:8091 ssh_username:root ? [('ns_1@10.112.190.102', 8091)] 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.102:8091 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.102:8091 is running 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.103:8091 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.103:8091 is running 2021-03-10 11:58:34 | INFO | MainProcess | test_thread | [basetestcase.setUp] Building docker image with java sdk client 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root] 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:58:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.029312288613303, 'mem_free': 2091716608, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:58:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:58:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:58:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_backup_restore_with_scheduled_partition_indexes ============== 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.101, nothing to delete 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.102, nothing to delete 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 10.112.190.103, nothing to delete 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.101:8091 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.101:8091 is running 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.102:8091 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.102:8091 is running 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.112.190.103:8091 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.112.190.103:8091 is running 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_backup_restore_with_scheduled_partition_indexes ============== 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-03-10 11:58:45 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.101 port:8091 ssh_username:root, nodes/self 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '181', 'memoryTotal': 2962632704, 'memoryFree': 2091716608, 'mcdMemoryReserved': 2260, 'mcdMemoryAllocated': 2260, 'status': 'healthy', 'hostname': '10.112.190.101:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4614-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1636, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.190.101', 'ip': '10.112.190.101', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 2825, 'curr_items': 0} 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 10.112.190.101 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1258 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,10.112.190.101,8091,['kv', 'index', 'n1ql']) 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.112.190.101: 8091:hostname=10.112.190.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-03-10 11:58:46 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://10.112.190.101:8091//node/controller/setupServices body: hostname=10.112.190.101&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.101:8091:port=8091&username=Administrator&password=password 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.101:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.102 port:8091 ssh_username:root, nodes/self 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '180', 'memoryTotal': 2962632704, 'memoryFree': 2538016768, 'mcdMemoryReserved': 2260, 'mcdMemoryAllocated': 2260, 'status': 'healthy', 'hostname': '10.112.190.102:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4614-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 980, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@10.112.190.102', 'ip': '10.112.190.102', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 2825, 'curr_items': 0} 2021-03-10 11:58:46 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://10.112.190.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1514 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-10 11:58:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.102:8091:port=8091&username=Administrator&password=password 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.102:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.112.190.103 port:8091 ssh_username:root, nodes/self 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '181', 'memoryTotal': 2962632704, 'memoryFree': 2519748608, 'mcdMemoryReserved': 2260, 'mcdMemoryAllocated': 2260, 'status': 'healthy', 'hostname': '10.112.190.103:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4614-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1892, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.190.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 2825, 'curr_items': 0} 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1514 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.112.190.103:8091:port=8091&username=Administrator&password=password 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:58:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 10.112.190.103:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-10 11:58:48 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-10 11:58:48 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.112.190.101 **** 2021-03-10 11:58:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-10 11:58:53 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-03-10 11:58:53 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-03-10 11:58:54 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.112.190.102:8091 to cluster 2021-03-10 11:58:54 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.112.190.102:8091 to this cluster @10.112.190.101:8091 2021-03-10 11:59:04 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-03-10 11:59:04 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-10 11:59:16 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@10.112.190.101,ns_1@10.112.190.102', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-03-10 11:59:16 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-03-10 11:59:16 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-10 11:59:16 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-03-10 11:59:26 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-03-10 11:59:26 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.036720037460327 sec 2021-03-10 11:59:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:59:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:59:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 10.112.190.101 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_backup_restore_with_scheduled_partition_indexes ============== 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 11:59:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:59:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:59:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.103 with username:root, attempt#1 of 5 2021-03-10 11:59:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.103 as root 2021-03-10 11:59:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:59:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.190.101:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.667731629392971, 'mem_free': 2179223552, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 10.112.190.102:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.329243353783231, 'mem_free': 2524049408, 'mem_total': 2962632704, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.190.101 cluster is healthy 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.190.101 is healthy 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.190.102 cluster is healthy 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.190.102 is healthy 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 10.112.190.103 cluster is healthy 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 10.112.190.103 is healthy 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings 2021-03-10 11:59:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-03-10 11:59:37 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set 2021-03-10 11:59:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 11:59:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 11:59:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 11:59:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 11:59:46 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2021-03-10 11:59:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 11:59:46 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:10.112.190.101 port:8091 ssh_username:root 2021-03-10 11:59:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2021-03-10 12:00:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 12:00:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 12:00:16 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:10.112.190.101 port:8091 ssh_username:root, ip:10.112.190.102 port:8091 ssh_username:root] 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: rm -rf /tmp/backup/ 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.102 with username:root, attempt#1 of 5 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.102 as root 2021-03-10 12:00:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.102: rm -rf /tmp/backup/ 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [rest_client.create_bucket] http://10.112.190.101:8091/pools/default/buckets with param: name=test_bucket_1&ramQuotaMB=200&authType=none&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [rest_client.create_bucket] 0.04 seconds to create bucket test_bucket_1 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX primary_idx_1 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:18 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+primary_idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.334357479s 2021-03-10 12:00:22 | ERROR | MainProcess | test_thread | [base_gsi.wait_until_indexes_online] No indexes are present, this check does not apply! 2021-03-10 12:00:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1_1_1 ON default:test_bucket_1(firstName,lastName) USING GSI WITH {'defer_build': False} 2021-03-10 12:00:22 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1_1_1+ON+default%3Atest_bucket_1%28firstName%2ClastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D 2021-03-10 12:00:26 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.234690527s 2021-03-10 12:00:26 | INFO | MainProcess | test_thread | [backup_and_restore.setUp] Enabling debug logs for Indexer 2021-03-10 12:00:26 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.settings.log_level': 'debug'} set 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.log_level': 'debug'} set 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_1_1_1 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_1_1_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 142.097497ms 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX primary_idx_1 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+primary_idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 90.539129ms 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [backup_and_restore._drop_indexes] 'status' 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 12:00:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_0 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_2 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_2 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_1 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_1 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_4 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_4 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_3 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_6 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_6 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_5 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_5 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_7 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_7 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_1 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_1+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_2 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_2+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_0 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_0+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_6 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_6+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_4 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_4+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_8 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_8 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_9 | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_9 ON default:test_bucket_1(firstName) PARTITION BY HASH (meta().id) USING GSI WITH {'nodes': ['10.112.190.101:8091', '10.112.190.102:8091'], 'defer_build': False} 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_3+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_7 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_7+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_5 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_5+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_8 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_8+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:27 | INFO | MainProcess | ThreadPoolExecutor-0_9 | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_9+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_6 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_6+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "a72a50f0-6643-4503-a890-887db7364887",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_6, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "117.396512ms","executionTime": "117.396318ms","resultCount": 0,"resultSize": 0,"serviceLoad": 250,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_2 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_2+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "fc51d31c-2d32-42d1-91f7-6d908f2c8e44",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_2, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "137.970996ms","executionTime": "137.944306ms","resultCount": 0,"resultSize": 0,"serviceLoad": 225,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_1 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_1+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "a23d860c-ed23-4200-ac10-c9506ae37152",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_1, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "155.932983ms","executionTime": "155.862558ms","resultCount": 0,"resultSize": 0,"serviceLoad": 200,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_0 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_0+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "abc09e72-6e0b-4118-b51a-8f93e1b7e69c",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_0, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "155.168666ms","executionTime": "155.144516ms","resultCount": 0,"resultSize": 0,"serviceLoad": 175,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_5 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_5+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "fff4d61d-2f16-47ab-82c2-a5952f288096",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_5, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "479.408172ms","executionTime": "358.762702ms","resultCount": 0,"resultSize": 0,"serviceLoad": 150,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_7 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_7+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "7149d899-8c07-4f6e-85ee-bae5af562696",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_7, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "563.014659ms","executionTime": "442.402304ms","resultCount": 0,"resultSize": 0,"serviceLoad": 125,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:28 | ERROR | MainProcess | ThreadPoolExecutor-0_4 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_4+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "c3183394-301d-4253-87a9-29be751ad667",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_4, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "644.244841ms","executionTime": "515.132946ms","resultCount": 0,"resultSize": 0,"serviceLoad": 100,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:29 | ERROR | MainProcess | ThreadPoolExecutor-0_9 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_9+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "175c45e2-300c-47cd-bb95-701b49d13b22",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_9, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.052299389s","executionTime": "534.03897ms","resultCount": 0,"resultSize": 0,"serviceLoad": 75,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:29 | ERROR | MainProcess | ThreadPoolExecutor-0_8 | [rest_client._http_request] POST http://10.112.190.101:8093/query?statement=CREATE+INDEX+idx_8+ON+default%3Atest_bucket_1%28firstName%29+PARTITION+BY+HASH+%28meta%28%29.id%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%2710.112.190.101%3A8091%27%2C+%2710.112.190.102%3A8091%27%5D%2C+%27defer_build%27%3A+False%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "1744adf2-b548-422c-873d-adf89104d6ce",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_8, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.111919371s","executionTime": "507.251885ms","resultCount": 0,"resultSize": 0,"serviceLoad": 50,"errorCount": 1}\n}\n' auth: Administrator:password 2021-03-10 12:00:32 | INFO | MainProcess | ThreadPoolExecutor-0_3 | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.56756405s 2021-03-10 12:00:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 1 secs. Wait till indexes available in stats ... 2021-03-10 12:00:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 12:00:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 12:00:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: /opt/couchbase/bin/cbbackupmgr config --archive /tmp/backup/ --repo backup_481790915 --disable-ft-indexes --disable-views --disable-data --disable-ft-alias --disable-eventing --disable-analytics --include-data test_bucket_1 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backup repository `backup_481790915` created successfully in archive `/tmp/backup/` 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: /opt/couchbase/bin/cbbackupmgr backup --archive /tmp/backup/ --repo backup_481790915 --cluster couchbase://10.112.190.101 --username Administrator --password password 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backing up to '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring bucket configuration for bucket 'test_bucket_1' 0 items / 0B 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [============================ ] 40.00% 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backing up to '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring bucket configuration for bucket 'test_bucket_1' 0 items / 0B 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [============================ ] 40.00% 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backing up to '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'test_bucket_1' 0 items / 0B 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [========================================== ] 60.00% 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backing up to '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 0 items / 0B 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backing up to '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 0 items / 0B 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backing up to '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Copied all data in 78ms (Avg. 0B/Sec) 0 items / 0B 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Transfer 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | -------- 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Status | Avg Transfer Rate | Started At | Finished At | Duration | 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Succeeded | 0B | Tue, 09 Mar 2021 22:30:34 -0800/s | Tue, 09 Mar 2021 22:30:34 -0800 | 100ms | 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backup completed successfully 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': 'abc09e72-6e0b-4118-b51a-8f93e1b7e69c', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_0, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '155.168666ms', 'executionTime': '155.144516ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': 'a23d860c-ed23-4200-ac10-c9506ae37152', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_1, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '155.932983ms', 'executionTime': '155.862558ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': 'fc51d31c-2d32-42d1-91f7-6d908f2c8e44', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_2, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '137.970996ms', 'executionTime': '137.944306ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] {'requestID': '08dd8c25-77de-4cba-b280-fde5663aa7dc', 'signature': None, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.56756405s', 'executionTime': '4.461812944s', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 25}} 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': 'c3183394-301d-4253-87a9-29be751ad667', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_4, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '644.244841ms', 'executionTime': '515.132946ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': 'fff4d61d-2f16-47ab-82c2-a5952f288096', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_5, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '479.408172ms', 'executionTime': '358.762702ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': 'a72a50f0-6643-4503-a890-887db7364887', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_6, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '117.396512ms', 'executionTime': '117.396318ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': '7149d899-8c07-4f6e-85ee-bae5af562696', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_7, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '563.014659ms', 'executionTime': '442.402304ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': '1744adf2-b548-422c-873d-adf89104d6ce', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_8, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '1.111919371s', 'executionTime': '507.251885ms 2021-03-10 12:00:34 | INFO | MainProcess | test_thread | [backup_and_restore.test_backup_restore_with_scheduled_partition_indexes] Raised Exception:CBQError: host 10.112.190.101: ERROR:{'requestID': '175c45e2-300c-47cd-bb95-701b49d13b22', 'signature': None, 'results': [], 'errors': [{'code': 5000, 'msg': 'GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_9, bucket test_bucket_1, scope _default, collection _default cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request..'}], 'status': 'errors', 'metrics': {'elapsedTime': '1.052299389s', 'executionTime': '534.03897ms' 2021-03-10 12:00:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_0 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:53 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_0+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 156.82422ms 2021-03-10 12:00:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_1 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:53 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_1+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 97.234808ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_2 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_2+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 97.081928ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_3 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_3+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 143.469744ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_4 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_4+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 116.192615ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_5 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_5+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 103.631935ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_6 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_6+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 99.412449ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_7 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_7+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 131.576642ms 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_8 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:54 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_8+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 112.622ms 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX idx_9 ON default:test_bucket_1 USING GSI 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+INDEX+idx_9+ON+default%3Atest_bucket_1+USING+GSI 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 134.012707ms 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [backup_and_restore._drop_indexes] 'status' 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 10.112.190.101 with username:root, attempt#1 of 5 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 10.112.190.101 as root 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-10 12:00:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.112.190.101: /opt/couchbase/bin/cbbackupmgr restore --archive /tmp/backup/ --repo backup_481790915 --cluster couchbase://10.112.190.101 --username Administrator --password password --force-updates --disable-ft-indexes --disable-views --disable-data --disable-ft-alias --disable-eventing --disable-analytics --include-data test_bucket_1 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Automatically creating scopes/collections for bucket 'test_bucke... 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [============== ] 20.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'test_bucket_1' 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [============================ ] 40.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'test_bucket_1' 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [============================ ] 40.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restoring backup '2021-03-09T22_30_34.453183056-08_00' 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Copied all data in 1.136s (Avg. 0B/Sec) 0 items / 0B 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Transfer 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | -------- 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Status | Avg Transfer Rate | Started At | Finished At | Duration | 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Succeeded | 0B | Tue, 09 Mar 2021 22:30:55 -0800/s | Tue, 09 Mar 2021 22:30:56 -0800 | 1.141s | 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Restore completed successfully 2021-03-10 12:00:56 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. wait to complete restore ... 2021-03-10 12:01:41 | INFO | MainProcess | test_thread | [backup_and_restore._verify_indexes] indexes before backup:[{'defnId': 4505063825399652122, 'instId': 4505063825399652122, 'name': 'idx_0', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 7402007915869846395, 'instId': 7402007915869846395, 'name': 'idx_1', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_1` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_1', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 10865078147842038199, 'instId': 10865078147842038199, 'name': 'idx_2', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_2` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_2', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 8772678105974664190, 'instId': 7175676540335679850, 'name': 'idx_3', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Ready', 'definition': 'CREATE INDEX `idx_3` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091', '10.112.190.102:8091'], 'completion': 100, 'progress': 100, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': {'10.112.190.101:8091': [6, 2, 4, 1], '10.112.190.102:8091': [5, 7, 3, 8]}, 'numReplica': 0, 'indexName': 'idx_3', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6738045941013073569, 'instId': 6738045941013073569, 'name': 'idx_4', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_4', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1498978878601761634, 'instId': 1498978878601761634, 'name': 'idx_5', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_5` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_5', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4690748906943597301, 'instId': 4690748906943597301, 'name': 'idx_6', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_6` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_6', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 13115121134956243556, 'instId': 13115121134956243556, 'name': 'idx_7', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_7` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_7', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 17673698091127721351, 'instId': 17673698091127721351, 'name': 'idx_8', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_8` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_8', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 4217190975989502695, 'instId': 4217190975989502695, 'name': 'idx_9', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Scheduled for Creation', 'definition': 'CREATE INDEX `idx_9` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': None, 'numReplica': 0, 'indexName': 'idx_9', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}] 2021-03-10 12:01:41 | INFO | MainProcess | test_thread | [backup_and_restore._verify_indexes] indexes after restore:[{'defnId': 16242893241750726445, 'instId': 660612050755839593, 'name': 'idx_0', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 6, 'partitionMap': {'10.112.190.102:8091': [3, 5, 4, 1, 8, 6]}, 'numReplica': 0, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 16242893241750726445, 'instId': 6044728028519799387, 'name': 'idx_0', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_0` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 2, 'partitionMap': {'10.112.190.101:8091': [2, 7]}, 'numReplica': 0, 'indexName': 'idx_0', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 18189353713448211841, 'instId': 5917328102157353444, 'name': 'idx_1', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_1` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 2, 'partitionMap': {'10.112.190.101:8091': [7, 3]}, 'numReplica': 0, 'indexName': 'idx_1', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 18189353713448211841, 'instId': 11629931823270063737, 'name': 'idx_1', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_1` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 6, 'partitionMap': {'10.112.190.102:8091': [4, 5, 6, 2, 1, 8]}, 'numReplica': 0, 'indexName': 'idx_1', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 17982622651762667137, 'instId': 15953081237883490187, 'name': 'idx_2', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_2` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 2, 'partitionMap': {'10.112.190.101:8091': [3, 4]}, 'numReplica': 0, 'indexName': 'idx_2', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 17982622651762667137, 'instId': 6887603169820752072, 'name': 'idx_2', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_2` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 6, 'partitionMap': {'10.112.190.102:8091': [7, 1, 8, 6, 2, 5]}, 'numReplica': 0, 'indexName': 'idx_2', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 7967482572777630371, 'instId': 3731508972810085593, 'name': 'idx_3', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_3` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091', '10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 8, 'partitionMap': {'10.112.190.101:8091': [6, 2, 4, 1], '10.112.190.102:8091': [5, 7, 3, 8]}, 'numReplica': 0, 'indexName': 'idx_3', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 11101094478883722885, 'instId': 4672209237811483985, 'name': 'idx_4', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 2, 'partitionMap': {'10.112.190.102:8091': [6, 7]}, 'numReplica': 0, 'indexName': 'idx_4', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 11101094478883722885, 'instId': 12369024114012467056, 'name': 'idx_4', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_4` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 6, 'partitionMap': {'10.112.190.101:8091': [5, 8, 3, 1, 2, 4]}, 'numReplica': 0, 'indexName': 'idx_4', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 16028543707449283390, 'instId': 11560743966438531437, 'name': 'idx_5', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_5` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 5, 'partitionMap': {'10.112.190.102:8091': [4, 5, 1, 2, 6]}, 'numReplica': 0, 'indexName': 'idx_5', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 16028543707449283390, 'instId': 2618305667069625765, 'name': 'idx_5', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_5` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 3, 'partitionMap': {'10.112.190.101:8091': [7, 8, 3]}, 'numReplica': 0, 'indexName': 'idx_5', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 9826611544347958531, 'instId': 9468164337502389383, 'name': 'idx_6', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_6` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 4, 'partitionMap': {'10.112.190.102:8091': [5, 3, 6, 4]}, 'numReplica': 0, 'indexName': 'idx_6', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 9826611544347958531, 'instId': 15174920743353360390, 'name': 'idx_6', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_6` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 4, 'partitionMap': {'10.112.190.101:8091': [1, 2, 8, 7]}, 'numReplica': 0, 'indexName': 'idx_6', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1492507314350803261, 'instId': 12290309310695954933, 'name': 'idx_7', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_7` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 2, 'partitionMap': {'10.112.190.102:8091': [8, 6]}, 'numReplica': 0, 'indexName': 'idx_7', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 1492507314350803261, 'instId': 2109228040172208188, 'name': 'idx_7', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_7` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 6, 'partitionMap': {'10.112.190.101:8091': [5, 4, 7, 1, 2, 3]}, 'numReplica': 0, 'indexName': 'idx_7', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 11078579307331446774, 'instId': 3369790023379452444, 'name': 'idx_8', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_8` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 2, 'partitionMap': {'10.112.190.102:8091': [4, 2]}, 'numReplica': 0, 'indexName': 'idx_8', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 11078579307331446774, 'instId': 3534122536194537713, 'name': 'idx_8', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_8` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 6, 'partitionMap': {'10.112.190.101:8091': [3, 6, 5, 7, 8, 1]}, 'numReplica': 0, 'indexName': 'idx_8', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6280843913381585571, 'instId': 7512000775622698269, 'name': 'idx_9', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_9` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.101:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 5, 'partitionMap': {'10.112.190.101:8091': [3, 5, 7, 2, 1]}, 'numReplica': 0, 'indexName': 'idx_9', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}, {'defnId': 6280843913381585571, 'instId': 5230682088840470787, 'name': 'idx_9', 'bucket': 'test_bucket_1', 'scope': '_default', 'collection': '_default', 'secExprs': ['`firstName`'], 'indexType': 'plasma', 'status': 'Created', 'definition': 'CREATE INDEX `idx_9` ON `test_bucket_1`(`firstName`) PARTITION BY hash((meta().`id`)) WITH { "defer_build":true, "nodes":[ "10.112.190.101:8091","10.112.190.102:8091" ], "num_partition":8 }', 'hosts': ['10.112.190.102:8091'], 'completion': 0, 'progress': 0, 'scheduled': False, 'partitioned': True, 'numPartition': 3, 'partitionMap': {'10.112.190.102:8091': [8, 6, 4]}, 'numReplica': 0, 'indexName': 'idx_9', 'replicaId': 0, 'stale': False, 'lastScanTime': 'NA'}]