INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'bucket_storage': 'couchstore', 'cluster_name': 'neo', 'force-updates': 'True', 'force_kill_memached': 'True', 'get-cbcollect-info': 'False', 'get-coredumps': 'False', 'ini': '/Users/giladkalchheim/work/ini/neo.ini', 'num_nodes': 4, 'sirius_url': 'http://172.23.120.103:4000', 'skip_log_scan': 'True', 'verify_unacked_bytes': 'True'} INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest.test_multiple_backups_merges,actions=backup_with_ops:1,backup:1,rebalance,backup_with_ops:1,backup:1,merge:1&4,nodes_in=1,document_type=binary,GROUP=P0,get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,skip_log_scan=True,force-updates=True,get-cbcollect-info=False,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 INFO:__main__:Logs folder: /Users/giladkalchheim/work/testrunner/logs/testrunner-23-Aug-29_13-51-26/test_1 Logs will be stored at /Users/giladkalchheim/work/testrunner/logs/testrunner-23-Aug-29_13-51-26/test_1 ./testrunner -i /Users/giladkalchheim/work/ini/neo.ini -p get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,skip_log_scan=True,force-updates=True,get-cbcollect-info=False,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 -t ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest.test_multiple_backups_merges,actions=backup_with_ops:1,backup:1,rebalance,backup_with_ops:1,backup:1,merge:1&4,nodes_in=1,document_type=binary,GROUP=P0,get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,skip_log_scan=True,force-updates=True,get-cbcollect-info=False,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 Test Input params: {'actions': 'backup_with_ops:1,backup:1,rebalance,backup_with_ops:1,backup:1,merge:1&4', 'nodes_in': '1', 'document_type': 'binary', 'GROUP': 'P0', 'get-coredumps': 'False', 'force_kill_memached': 'True', 'verify_unacked_bytes': 'True', 'skip_log_scan': 'True', 'force-updates': 'True', 'get-cbcollect-info': 'False', 'bucket_storage': 'couchstore', 'sirius_url': 'http://172.23.120.103:4000', 'ini': '/Users/giladkalchheim/work/ini/neo.ini', 'cluster_name': 'neo', 'num_nodes': 4, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/Users/giladkalchheim/work/testrunner/logs/testrunner-23-Aug-29_13-51-26/test_1'} Run before suite setup for ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest.test_multiple_backups_merges -->before_suite_name:ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest.suite_setUp,suite: ]> suite_setUp (ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest) ... 2023-08-29 13:51:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:51:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:51:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:51:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:51:36 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:51:37 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 13:51:37 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2023-08-29 13:51:38 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2023-08-29 13:51:38 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.137.160:8091 => {'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.646236362574512, 'mem_free': 3288936448, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 13:51:38 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2023-08-29 13:51:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2023-08-29 13:51:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.160, nothing to delete 2023-08-29 13:51:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 13:51:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 13:51:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 13:51:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2023-08-29 13:51:42 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.137.160:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2023-08-29 13:51:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2023-08-29 13:51:43 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:51:44 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.160:8091 2023-08-29 13:51:44 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:51:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.160:8091 is running 2023-08-29 13:51:45 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.136.166:8091 2023-08-29 13:51:45 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:51:46 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.136.166:8091 is running 2023-08-29 13:51:46 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.161:8091 2023-08-29 13:51:46 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:51:46 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.161:8091 is running 2023-08-29 13:51:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.159:8091 2023-08-29 13:51:47 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:51:47 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.159:8091 is running 2023-08-29 13:51:47 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2023-08-29 13:51:47 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2023-08-29 13:51:47 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2023-08-29 13:51:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:51:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:51:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:51:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:51:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: rm -rf /cbqe3043/entbackup 2023-08-29 13:51:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:51:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.160, nothing to delete 2023-08-29 13:51:58 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 13:51:59 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 13:51:59 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:52:01 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:52:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:52:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:52:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 13:52:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:52:05 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:52:06 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2023-08-29 13:52:07 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.137.160 port:8091 ssh_username:root, nodes/self 2023-08-29 13:52:07 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11464', 'memoryTotal': 4201684992, 'memoryFree': 3291553792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.137.160:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@172.23.137.160', 'ip': '172.23.137.160', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:52:08 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2023-08-29 13:52:08 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.137.160 2023-08-29 13:52:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2023-08-29 13:52:08 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2023-08-29 13:52:08 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.137.160 2023-08-29 13:52:08 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2023-08-29 13:52:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1379 2023-08-29 13:52:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.137.160,8091,['kv', 'index', 'n1ql', 'fts', 'eventing', 'backup']) 2023-08-29 13:52:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.137.160: 8091:hostname=172.23.137.160&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing%2Cbackup 2023-08-29 13:52:09 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.137.160:8091//node/controller/setupServices body: hostname=172.23.137.160&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password 2023-08-29 13:52:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2023-08-29 13:52:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:52:09 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.137.160:8091:port=8091&username=Administrator&password=password 2023-08-29 13:52:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:52:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:52:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:52:14 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:52:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:52:19 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.160:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:19 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.160:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:52:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.136.166 port:8091 ssh_username:root, nodes/self 2023-08-29 13:52:21 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11477', 'memoryTotal': 4201684992, 'memoryFree': 3538702336, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.136.166:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.136.166', 'ip': '172.23.136.166', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:52:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.136.166:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-08-29 13:52:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2023-08-29 13:52:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:52:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.136.166:8091:port=8091&username=Administrator&password=password 2023-08-29 13:52:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:52:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.136.166 with username:root, attempt#1 of 5 2023-08-29 13:52:24 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.136.166 as root 2023-08-29 13:52:27 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:52:31 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:52:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.136.166: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:52:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:52:31 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:52:32 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.136.166:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:32 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.136.166:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:33 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:52:34 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.137.161 port:8091 ssh_username:root, nodes/self 2023-08-29 13:52:34 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11492', 'memoryTotal': 4201684992, 'memoryFree': 3441008640, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.137.161:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.137.161', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:52:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2023-08-29 13:52:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:52:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.137.161:8091:port=8091&username=Administrator&password=password 2023-08-29 13:52:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:52:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.161 with username:root, attempt#1 of 5 2023-08-29 13:52:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.161 as root 2023-08-29 13:52:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:52:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:52:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.161: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:52:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:52:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:52:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.161:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.161:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:45 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:52:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.137.159 port:8091 ssh_username:root, nodes/self 2023-08-29 13:52:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11503', 'memoryTotal': 4201684992, 'memoryFree': 3447902208, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.137.159:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.137.159', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:52:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2023-08-29 13:52:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:52:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.137.159:8091:port=8091&username=Administrator&password=password 2023-08-29 13:52:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:52:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:52:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:52:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:52:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:52:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:52:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:52:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:52:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.159:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.159:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:52:58 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:52:59 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.137.160 **** 2023-08-29 13:53:02 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2023-08-29 13:53:07 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2023-08-29 13:53:08 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2023-08-29 13:53:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:53:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:53:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:53:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:53:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.137.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:53:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:53:17 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:53:17 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.137.160 2023-08-29 13:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.137.160:8091/pools/default/buckets with param: name=default&ramQuotaMB=1379&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2023-08-29 13:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.37 seconds to create bucket default 2023-08-29 13:53:20 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.137.160 to accept set ops 2023-08-29 13:53:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 13:53:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 13:56:22 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1379 2023-08-29 13:56:22 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2023-08-29 13:56:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2023-08-29 13:56:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.137.160:8091 => {'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.402268669982148, 'mem_free': 3252576256, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 13:56:23 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2023-08-29 13:56:23 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 296.0872850418091 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52 100 20 100 32 59 95 --:--:-- --:--:-- --:--:-- 159 2023-08-29 13:56:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:56:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:56:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:56:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:56:32 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 13:56:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:56:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:56:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:56:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:56:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: nproc 2023-08-29 13:56:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:56:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:56:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:56:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:56:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:56:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:56:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:56:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:57:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:57:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2023-08-29 13:57:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.137.160:8091 => {'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.756918907973712, 'mem_free': 3248488448, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 13:57:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2023-08-29 13:57:01 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2023-08-29 13:57:03 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.137.160 2023-08-29 13:57:04 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2023-08-29 13:57:04 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.137.160 existing buckets : [] 2023-08-29 13:57:04 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.137.160 2023-08-29 13:57:05 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 13:57:05 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 13:57:06 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 13:57:06 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2023-08-29 13:57:07 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.137.160:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2023-08-29 13:57:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2023-08-29 13:57:07 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:09 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.160:8091 2023-08-29 13:57:09 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:10 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.160:8091 is running 2023-08-29 13:57:10 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.136.166:8091 2023-08-29 13:57:10 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:10 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.136.166:8091 is running 2023-08-29 13:57:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.161:8091 2023-08-29 13:57:11 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.161:8091 is running 2023-08-29 13:57:11 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.159:8091 2023-08-29 13:57:11 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.159:8091 is running 2023-08-29 13:57:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2023-08-29 13:57:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2023-08-29 13:57:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2023-08-29 13:57:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:57:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:57:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:57:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:57:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: rm -rf /cbqe3043/entbackup 2023-08-29 13:57:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:57:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.160, nothing to delete 2023-08-29 13:57:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 13:57:23 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 13:57:23 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:25 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:26 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:26 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:27 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 13:57:28 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:29 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:30 | INFO | MainProcess | MainThread | [enterprise_backup_restore_base.clean_up] delete dir /tmp/backuprestore172.23.137.160 ok Exception ignored in: Traceback (most recent call last): File "pytests/ent_backup_restore/enterprise_backup_merge_tests.py", line 17, in __del__ self.objstore_provider.__del__() AttributeError: 'NoneType' object has no attribute '__del__' ---------------------------------------------------------------------- Ran 1 test in 362.965s OK -->result: test_multiple_backups_merges (ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest) ... 2023-08-29 13:57:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:57:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:57:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:57:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:57:38 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:39 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 13:57:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_multiple_backups_merges============== 2023-08-29 13:57:40 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2023-08-29 13:57:40 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.137.160:8091 => {'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.414311505589954, 'mem_free': 3285045248, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 13:57:40 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2023-08-29 13:57:40 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_multiple_backups_merges ============== 2023-08-29 13:57:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.160, nothing to delete 2023-08-29 13:57:42 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 13:57:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 13:57:44 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 13:57:44 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2023-08-29 13:57:45 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.137.160:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2023-08-29 13:57:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2023-08-29 13:57:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.160:8091 2023-08-29 13:57:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.160:8091 is running 2023-08-29 13:57:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.136.166:8091 2023-08-29 13:57:47 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.136.166:8091 is running 2023-08-29 13:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.161:8091 2023-08-29 13:57:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.161:8091 is running 2023-08-29 13:57:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.159:8091 2023-08-29 13:57:49 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:57:49 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.159:8091 is running 2023-08-29 13:57:49 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_multiple_backups_merges ============== 2023-08-29 13:57:49 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2023-08-29 13:57:49 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2023-08-29 13:57:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:57:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:57:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:57:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:57:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: rm -rf /cbqe3043/entbackup 2023-08-29 13:57:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:57:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.160, nothing to delete 2023-08-29 13:58:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 13:58:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 13:58:01 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:58:02 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:58:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:58:04 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:58:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 13:58:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:58:07 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 13:58:07 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2023-08-29 13:58:08 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.137.160 port:8091 ssh_username:root, nodes/self 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11824', 'memoryTotal': 4201684992, 'memoryFree': 3289534464, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.137.160:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@172.23.137.160', 'ip': '172.23.137.160', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.137.160 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.137.160 2023-08-29 13:58:09 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2023-08-29 13:58:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1379 2023-08-29 13:58:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.137.160,8091,['kv', 'index', 'n1ql', 'fts', 'eventing', 'backup']) 2023-08-29 13:58:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.137.160: 8091:hostname=172.23.137.160&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing%2Cbackup 2023-08-29 13:58:10 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.137.160:8091//node/controller/setupServices body: hostname=172.23.137.160&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Ceventing%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password 2023-08-29 13:58:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2023-08-29 13:58:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:58:10 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.137.160:8091:port=8091&username=Administrator&password=password 2023-08-29 13:58:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:58:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:58:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:58:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:58:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:58:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:58:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:58:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:58:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.160:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.160:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:21 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:58:22 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.136.166 port:8091 ssh_username:root, nodes/self 2023-08-29 13:58:22 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11838', 'memoryTotal': 4201684992, 'memoryFree': 3530747904, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.136.166:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.136.166', 'ip': '172.23.136.166', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:58:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2023-08-29 13:58:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:58:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.136.166:8091:port=8091&username=Administrator&password=password 2023-08-29 13:58:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:58:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.136.166 with username:root, attempt#1 of 5 2023-08-29 13:58:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.136.166 as root 2023-08-29 13:58:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:58:31 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:58:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.136.166: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:58:32 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:58:32 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:58:32 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.136.166:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:33 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.136.166:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:33 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:58:34 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.137.161 port:8091 ssh_username:root, nodes/self 2023-08-29 13:58:35 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11853', 'memoryTotal': 4201684992, 'memoryFree': 3442266112, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.137.161:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.137.161', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:58:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2023-08-29 13:58:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:58:35 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.137.161:8091:port=8091&username=Administrator&password=password 2023-08-29 13:58:36 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:58:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.161 with username:root, attempt#1 of 5 2023-08-29 13:58:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.161 as root 2023-08-29 13:58:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:58:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:58:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.161: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:58:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:58:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:58:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.161:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.161:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:45 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:58:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.137.159 port:8091 ssh_username:root, nodes/self 2023-08-29 13:58:47 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '11864', 'memoryTotal': 4201684992, 'memoryFree': 3444375552, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.137.159:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5928-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.137.159', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4007, 'curr_items': 0} 2023-08-29 13:58:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2023-08-29 13:58:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2023-08-29 13:58:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.137.159:8091:port=8091&username=Administrator&password=password 2023-08-29 13:58:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2023-08-29 13:58:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 13:58:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 13:58:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:58:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:58:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:58:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:58:58 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:58:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.159:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:58 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.137.159:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). 2023-08-29 13:58:59 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2023-08-29 13:59:00 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.137.160 **** 2023-08-29 13:59:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2023-08-29 13:59:08 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2023-08-29 13:59:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2023-08-29 13:59:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 13:59:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 13:59:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 13:59:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 13:59:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-08-29 13:59:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 13:59:18 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2023-08-29 13:59:18 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.137.160 2023-08-29 13:59:20 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.137.160:8091/pools/default/buckets with param: name=default&ramQuotaMB=1379&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2023-08-29 13:59:21 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.37 seconds to create bucket default 2023-08-29 13:59:21 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.137.160 to accept set ops 2023-08-29 13:59:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 13:59:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:02:24 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1379 2023-08-29 14:02:24 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_multiple_backups_merges ============== 2023-08-29 14:02:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2023-08-29 14:02:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.137.160:8091 => {'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.403650668324122, 'mem_free': 3254341632, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 14:02:26 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2023-08-29 14:02:26 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 295.768061876297 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52 100 20 100 32 58 94 --:--:-- --:--:-- --:--:-- 155 2023-08-29 14:02:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:02:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:02:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:02:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:02:35 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:02:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:02:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:02:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:02:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:02:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: nproc 2023-08-29 14:02:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:02:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:02:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:02:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:02:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:02:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:02:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:02:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:03:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:03:04 | INFO | MainProcess | test_thread | [enterprise_backup_merge_tests.test_multiple_backups_merges] *** start to load items to all buckets 2023-08-29 14:03:04 | INFO | MainProcess | test_thread | [enterprise_backup_merge_tests.test_multiple_backups_merges] Disable compaction to speed up dgm 2023-08-29 14:03:04 | INFO | MainProcess | test_thread | [rest_client.disable_auto_compaction] Disable autocompaction in cluster-wide setting 2023-08-29 14:03:07 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:12 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:12 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:12 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:12 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:15 | INFO | MainProcess | test_thread | [enterprise_backup_merge_tests.test_multiple_backups_merges] *** done to load items to all buckets 2023-08-29 14:03:16 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:03:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:03:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:03:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:03:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:03:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: rm -rf /tmp/entbackup_172.23.137.160 2023-08-29 14:03:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:03:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: /opt/couchbase/bin//cbbackupmgr config --archive /tmp/entbackup_172.23.137.160 --repo backup 2023-08-29 14:03:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:03:25 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Backup repository `backup` created successfully in archive `/tmp/entbackup_172.23.137.160` 2023-08-29 14:03:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:03:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:03:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:03:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:03:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -R /tmp/entbackup_172.23.137.160/backup 2023-08-29 14:03:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:03:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:03:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:03:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:03:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:03:45 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_create_validate] Expected Backup Meta data json and Actual Backup Meta data json are equal Backup create validation success. 2023-08-29 14:03:45 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Performing backup_with_ops action for 1th time 2023-08-29 14:03:47 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:03:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:03:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:03:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:04:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:04:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:04:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:04:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:04:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:04:12 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:04:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr /tmp/entbackup_172.23.137.160/backup/ | grep -P '^\d{4}-\d{2}-\d{2}' | tail -1 2023-08-29 14:04:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:04:13 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster] Finished taking backup with args: backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:04:13 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:04:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:04:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:04:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:04:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:04:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -lR /tmp/entbackup_172.23.137.160/backup 2023-08-29 14:04:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:04:24 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster_validate] Expected Directory Structure and Actual Directory Structure are equal 2023-08-29 14:04:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:04:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:04:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:04:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:04:33 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:04:36 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:04:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:04:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:04:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:04:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:04:51 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:04:53 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:04:59 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_with_ops] [, , ] 2023-08-29 14:06:45 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Finished backup_with_ops action for 1th time. 2023-08-29 14:06:45 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Performing backup action for 1th time 2023-08-29 14:06:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:06:46 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:06:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:06:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:06:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:06:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:06:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr /tmp/entbackup_172.23.137.160/backup/ | grep -P '^\d{4}-\d{2}-\d{2}' | tail -1 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster] Finished taking backup with args: backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:07:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:07:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:07:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:07:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:07:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -lR /tmp/entbackup_172.23.137.160/backup 2023-08-29 14:07:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:07:12 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster_validate] Expected Directory Structure and Actual Directory Structure are equal 2023-08-29 14:07:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:07:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:07:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:07:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:07:22 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:07:24 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:07:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:07:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:07:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:07:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:07:41 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:07:44 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:07:50 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Finished backup action for 1th time. 2023-08-29 14:07:50 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Performing rebalance action for 1th time 2023-08-29 14:07:53 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.136.166:8091 to cluster 2023-08-29 14:07:54 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.136.166:18091 to this cluster @172.23.137.160:8091 2023-08-29 14:08:05 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.68 seconds 2023-08-29 14:08:05 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2023-08-29 14:08:18 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.136.166 not part of cluster inactiveAdded 2023-08-29 14:08:19 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.136.166,ns_1@172.23.137.160', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-08-29 14:08:20 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2023-08-29 14:08:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 53.00 % 2023-08-29 14:08:30 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 53.00% 2023-08-29 14:08:51 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2023-08-29 14:08:51 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 31.43562388420105 sec 2023-08-29 14:08:51 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Finished rebalance action for 1th time. 2023-08-29 14:08:51 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Performing backup_with_ops action for 1th time 2023-08-29 14:08:53 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.136.166:11210 default 2023-08-29 14:08:56 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:09:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.136.166:11210 default 2023-08-29 14:09:04 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:09:09 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.136.166:11210 default 2023-08-29 14:09:12 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.137.160:11210 default 2023-08-29 14:09:15 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:09:16 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:09:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:09:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:09:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:09:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:09:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr /tmp/entbackup_172.23.137.160/backup/ | grep -P '^\d{4}-\d{2}-\d{2}' | tail -1 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster] Finished taking backup with args: backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:09:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:09:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:09:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:09:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:09:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -lR /tmp/entbackup_172.23.137.160/backup 2023-08-29 14:09:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:09:40 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster_validate] Expected Directory Structure and Actual Directory Structure are equal 2023-08-29 14:09:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.136.166 with username:root, attempt#1 of 5 2023-08-29 14:09:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.136.166 as root 2023-08-29 14:09:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:09:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:09:49 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:09:52 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:10:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:10:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:10:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:10:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:10:10 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:10:13 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:10:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.136.166 with username:root, attempt#1 of 5 2023-08-29 14:10:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.136.166 as root 2023-08-29 14:10:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:10:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:10:32 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:10:35 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:10:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:10:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:10:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:10:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:10:52 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:10:55 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:11:05 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_with_ops] [, , ] 2023-08-29 14:11:57 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Finished backup_with_ops action for 1th time. 2023-08-29 14:11:57 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Performing backup action for 1th time 2023-08-29 14:11:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:11:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:11:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:12:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:12:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:12:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:12:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr /tmp/entbackup_172.23.137.160/backup/ | grep -P '^\d{4}-\d{2}-\d{2}' | tail -1 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster] Finished taking backup with args: backup --archive /tmp/entbackup_172.23.137.160 --repo backup --cluster http://172.23.137.160:8091 --username Administrator --password password --no-progress-bar 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base._check_output] Found 'Backup completed successfully' in CLI output 2023-08-29 14:12:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:12:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:12:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:12:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:12:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -lR /tmp/entbackup_172.23.137.160/backup 2023-08-29 14:12:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:12:23 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_cluster_validate] Expected Directory Structure and Actual Directory Structure are equal 2023-08-29 14:12:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.136.166 with username:root, attempt#1 of 5 2023-08-29 14:12:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.136.166 as root 2023-08-29 14:12:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:12:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:12:32 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:12:35 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:12:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:12:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:12:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:12:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:12:53 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:12:56 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:13:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.136.166 with username:root, attempt#1 of 5 2023-08-29 14:13:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.136.166 as root 2023-08-29 14:13:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:13:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:13:15 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:13:17 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:13:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:13:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:13:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:13:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:13:35 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:13:38 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:13:48 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Finished backup action for 1th time. 2023-08-29 14:13:48 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.do_backup_merge_actions] Performing merge action for 1th time 2023-08-29 14:13:48 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_merge] backups before merge: ['2023-08-29T06_04_08.972606547-07_00', '2023-08-29T06_06_56.078501634-07_00', '2023-08-29T06_09_25.200651154-07_00', '2023-08-29T06_12_08.27655981-07_00'] 2023-08-29 14:13:48 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_merge] number_of_backups_taken before merge: 4 2023-08-29 14:13:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:13:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:13:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:13:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:13:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: /opt/couchbase/bin//cbbackupmgr merge --archive /tmp/entbackup_172.23.137.160 --repo backup --start 2023-08-29T06_04_08.972606547-07_00 --end 2023-08-29T06_12_08.27655981-07_00 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring eventing metadata 0 items / 0B 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring bucket configuration for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Analytics metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text indexes for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring views definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Checking for data movement restrictions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Deciding which key value data to transfer for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value data for buc... at 0B/s (about 0s remaining) 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key valu... at 20.43KiB/s (about 6s remaining) 135 items / 20.54KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key valu... at 21.85KiB/s (about 4s remaining) 312 items / 46.66KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key valu... at 23.53KiB/s (about 3s remaining) 543 items / 80.80KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key val... at 24.83KiB/s (about 1s remaining) 774 items / 115.14KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key va... at 25.82KiB/s (about 0s remaining) 1001 items / 149.03KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00'====== ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 1117 items / 165.64KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00'=============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 1117 items / 165.64KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (1/4) Merging backup '2023-08-29T06_04_08.972606547-07_00'=============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Copied all data in 5.484s (Avg. 33.13KiB/Sec) 1117 items / 165.64KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Transfer 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | -------- 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Status | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Succeeded | 33.13KiB/s | Tue, 29 Aug 2023 06:13:57 -0700 | Tue, 29 Aug 2023 06:14:02 -0700 | 5.482s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Bucket 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | ------ 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | default | Succeeded | 165.64KiB | 33.13KiB/s | Tue, 29 Aug 2023 06:13:57 -0700 | Tue, 29 Aug 2023 06:14:02 -0700 | 5.27s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Mutations | Deletions | Expirations | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | --------- | --------- | ----------- | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 1052 | 0 | 0 | 65 | 0 | 0 | 0 | 0 | 0 | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring eventing metadata 0 items / 0B 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring bucket configuration for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Analytics metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring views definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Checking for data movement restrictions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Deciding which key value data to transfer for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value dat... at 25.82KiB/s (about 7m31s remaining) 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key val... at 22.05KiB/s (about 25s remaining) 175 items / 20.70KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key val... at 19.85KiB/s (about 11s remaining) 439 items / 49.83KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key valu... at 18.56KiB/s (about 5s remaining) 723 items / 80.85KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key val... at 17.80KiB/s (about 2s remaining) 995 items / 111.17KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key va... at 17.37KiB/s (about 0s remaining) 1271 items / 141.36KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00'============= ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 1274 items / 141.66KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00'=============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 1274 items / 141.66KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (2/4) Merging backup '2023-08-29T06_06_56.078501634-07_00'=============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Copied all data in 10.503s (Avg. 14.17KiB/Sec) 1274 items / 141.66KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Transfer 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | -------- 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Status | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Succeeded | 28.33KiB/s | Tue, 29 Aug 2023 06:14:02 -0700 | Tue, 29 Aug 2023 06:14:07 -0700 | 5.017s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Bucket 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | ------ 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | default | Succeeded | 141.66KiB | 35.42KiB/s | Tue, 29 Aug 2023 06:14:02 -0700 | Tue, 29 Aug 2023 06:14:07 -0700 | 4.766s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Mutations | Deletions | Expirations | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | --------- | --------- | ----------- | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 439 | 0 | 0 | 835 | 0 | 0 | 0 | 0 | 0 | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Analytics metadata 0 items / 0B 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring eventing metadata 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring bucket configuration for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Analytics metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text indexes for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Checking for data movement restrictions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Deciding which key value data to transfer for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value dat... at 17.37KiB/s (about 14m7s remaining) 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value... at 15.54KiB/s (about 2s remaining) 64 items / 25.14KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00' ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key valu... at 13.61KiB/s (about 0s remaining) 197 items / 49.64KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00'============ ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 212 items / 52.26KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00'=============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 212 items / 52.26KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (3/4) Merging backup '2023-08-29T06_09_25.200651154-07_00'=============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Copied all data in 12.619s (Avg. 4.35KiB/Sec) 212 items / 52.26KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Transfer 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | -------- 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Status | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Succeeded | 26.13KiB/s | Tue, 29 Aug 2023 06:14:07 -0700 | Tue, 29 Aug 2023 06:14:09 -0700 | 2.115s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Bucket 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | ------ 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | default | Succeeded | 52.26KiB | 52.26KiB/s | Tue, 29 Aug 2023 06:14:07 -0700 | Tue, 29 Aug 2023 06:14:09 -0700 | 1.866s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Mutations | Deletions | Expirations | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | --------- | --------- | ----------- | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 212 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Analytics metadata 0 items / 0B 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text index aliases 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring bucket configuration for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring full text indexes for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring GSI index definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring views definitions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring Query metadata for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Checking for data movement restrictions for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Deciding which key value data to transfer for bucket 'default' 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value da... at 13.61KiB/s (about 16m58s remaining) 0 items / 0B8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00' ]2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value... at 11.15KiB/s (about 3s remaining) 53 items / 30.49KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00'= ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Transferring key value... at 8.69KiB/s (about 1s remaining) 249 items / 61.33KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00'========= ] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 388 items / 82.86KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00'==============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 388 items / 82.86KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] (4/4) Merging backup '2023-08-29T06_12_08.27655981-07_00'==============] 12023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Copied all data in 15.228s (Avg. 5.52KiB/Sec) 388 items / 82.86KiB8-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [======================================================================] 100.00% 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Transfer 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | -------- 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Status | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Succeeded | 41.43KiB/s | Tue, 29 Aug 2023 06:14:09 -0700 | Tue, 29 Aug 2023 06:14:12 -0700 | 2.608s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Bucket 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | ------ 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | default | Succeeded | 82.86KiB | 41.43KiB/s | Tue, 29 Aug 2023 06:14:10 -0700 | Tue, 29 Aug 2023 06:14:12 -0700 | 2.297s | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Mutations | Deletions | Expirations | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | --------- | --------- | ----------- | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] | 388 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Merge completed successfully 2023-08-29 14:14:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr /tmp/entbackup_172.23.137.160/backup | tail -1 2023-08-29 14:14:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:14:13 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_merge] backups after merge: ['2023-08-29T06_12_08.27655981-07_00'] 2023-08-29 14:14:13 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.backup_merge] number_of_backups_taken after merge: 1 2023-08-29 14:14:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:14:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:14:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:14:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:14:22 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise ---- Collecting data in backup repo 2023-08-29 14:14:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr /tmp/entbackup_172.23.137.160/backup | tail -1 2023-08-29 14:14:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:14:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: ls -tr --group-directories-first /tmp/entbackup_172.23.137.160/backup/2023-08-29T06_12_08.27655981-07_00/default-5c1738ab7a84a40e82b202d90991a8ff/data | grep index 2023-08-29 14:14:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root ---- Done extract data from backup files in backup repo of bucket default ---- total files with data in bucket default are 682 2023-08-29 14:20:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.160 with username:root, attempt#1 of 5 2023-08-29 14:20:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.160 as root 2023-08-29 14:20:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:21:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:21:01 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2023-08-29 14:21:03 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2023-08-29 14:21:13 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] *** Compare data in bucket and in backup file of bucket default *** 2023-08-29 14:21:13 | INFO | MainProcess | test_thread | [rebalance_helper.wait_for_stats] waiting for bucket default stat : ep_queue_size to match 0 on 172.23.136.166 2023-08-29 14:21:14 | INFO | MainProcess | test_thread | [rebalance_helper.wait_for_stats] ep_queue_size : 0 2023-08-29 14:21:14 | INFO | MainProcess | test_thread | [rebalance_helper.wait_for_stats] waiting for bucket default stat : ep_queue_size to match 0 on 172.23.137.160 2023-08-29 14:21:15 | INFO | MainProcess | test_thread | [rebalance_helper.wait_for_stats] ep_queue_size : 0 2023-08-29 14:21:16 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.1-5928-enterprise 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup86. bucket: "b'b""b\'ent-backup-86aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaVaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'A\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'K\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-86aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaVaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'A'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'K'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup8. bucket: "b'b""b\'ent-backup-8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'N\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaUaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'Y\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'N'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaUaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'Y'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup78. bucket: "b'b""b\'ent-backup-78aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'Q\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'Z\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-78aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'Q'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'Z'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup95. bucket: "b'b""b\'ent-backup-95aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'I\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'H\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-95aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'I'b"aaaaaaaaaaaaaaaaaaaaaaaaaaAaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'H'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup49. bucket: "b'b""b\'ent-backup-49aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'O\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'H\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaOaaaaaaa\'""'" != b"b'ent-backup-49aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'O'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'H'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaOaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup83. bucket: "b'b""b\'ent-backup-83aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'T\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'S\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaNaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-83aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'T'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'S'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaNaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup90. bucket: "b'b""b\'ent-backup-90aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'G\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaLaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'P\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-90aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'G'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaLaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'P'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup43. bucket: "b'b""b\'ent-backup-43aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaPaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'D\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'M\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-43aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaPaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'D'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'M'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup93. bucket: "b'b""b\'ent-backup-93aaaaaaa\'b\'Z\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'J\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaPaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-93aaaaaaa'b'Z'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'J'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaPaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup68. bucket: "b'b""b\'\'b\'H\'b\'nt-backup-68aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'I\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b''b'H'b'nt-backup-68aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'I'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup38. bucket: "b'b""b\'ent-backup-38aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'C\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'S\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaPaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-38aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'C'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'S'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaPaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup59. bucket: "b'b""b\'ent-backup-59aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaKaaa""b\'T\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'F\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-59aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaKaaa"b'T'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'F'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup10. bucket: "b'b""b\'ent-backup-10aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaGaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'O\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'R\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-10aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaGaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'O'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'R'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup45. bucket: "b'b""b\'ent-backup-45aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'W\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'J\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-45aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'W'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'J'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup55. bucket: "b'b""b\'ent-backup-55aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'Y\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaHaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'N\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-55aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'Y'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaHaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'N'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup46. bucket: "b'b""b\'ent-backup-46aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'Z\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'R\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaFaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-46aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'Z'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'R'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaFaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup7. bucket: "b'b""b\'ent-backup-7aa""b\'N\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'D\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaKaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-7aa"b'N'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'D'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaKaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup89. bucket: "b'b""b\'ent-backup-8Maaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'T\'b""aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'B\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-8Maaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'T'b"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'B'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup21. bucket: "b'b""b\'ent-backup-21aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'b\'J\'b\'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'K\'b""aaaaaaaaaaaJaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-backup-21aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'b'J'b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'K'b"aaaaaaaaaaaJaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'" file 2023-08-29 14:21:16 | ERROR | MainProcess | test_thread | [enterprise_backup_restore_base.validate_backup_data] Data does not match at key ent-backup64. bucket: "b'b""b\'ent-ba\'b\'O\'b\'kup-64aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa""b\'W\'b""aaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaa\'""'" != b"b'ent-ba'b'O'b'kup-64aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"b'W'b"aaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaa'" file Data not match in backup bucket default 2023-08-29 14:21:17 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2023-08-29 14:21:17 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.136.166:8091 => {'services': ['kv'], 'cpu_utilization': 1.296823633745083, 'mem_free': 3438673920, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 14:21:17 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.137.160:8091 => {'services': ['backup', 'eventing', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.961370327995423, 'mem_free': 3192987648, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2023-08-29 14:21:17 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2023-08-29 14:21:17 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_multiple_backups_merges ============== 2023-08-29 14:21:19 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.137.160 2023-08-29 14:21:20 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2023-08-29 14:21:20 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.137.160 existing buckets : [] 2023-08-29 14:21:20 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.137.160 2023-08-29 14:21:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 14:21:21 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 14:21:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 14:21:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2023-08-29 14:21:23 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.137.160:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2023-08-29 14:21:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2023-08-29 14:21:23 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:21:25 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2023-08-29 14:21:25 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.136.166,ns_1@172.23.137.160', 'ejectedNodes': 'ns_1@172.23.136.166', 'user': 'Administrator', 'password': 'password'} 2023-08-29 14:21:26 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2023-08-29 14:21:26 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 79.00 % 2023-08-29 14:21:47 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 21.03 seconds 2023-08-29 14:21:47 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2023-08-29 14:21:57 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.137.160 port:8091 ssh_username:root ? [('ns_1@172.23.136.166', 8091)] 2023-08-29 14:21:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.160:8091 2023-08-29 14:21:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:21:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.160:8091 is running 2023-08-29 14:21:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.136.166:8091 2023-08-29 14:21:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:21:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.136.166:8091 is running 2023-08-29 14:21:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.161:8091 2023-08-29 14:21:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:21:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.161:8091 is running 2023-08-29 14:22:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.137.159:8091 2023-08-29 14:22:00 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.137.159:8091 is running 2023-08-29 14:22:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_multiple_backups_merges ============== 2023-08-29 14:22:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2023-08-29 14:22:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2023-08-29 14:22:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.137.159 with username:root, attempt#1 of 5 2023-08-29 14:22:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.137.159 as root 2023-08-29 14:22:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-08-29 14:22:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-08-29 14:22:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.137.159: rm -rf /cbqe3043/entbackup 2023-08-29 14:22:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2023-08-29 14:22:10 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.160, nothing to delete 2023-08-29 14:22:11 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.136.166, nothing to delete 2023-08-29 14:22:12 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.161, nothing to delete 2023-08-29 14:22:12 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:14 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:15 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:15 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:16 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.137.159, nothing to delete 2023-08-29 14:22:17 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:18 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2023-08-29 14:22:19 | INFO | MainProcess | test_thread | [enterprise_backup_restore_base.clean_up] delete dir /tmp/backuprestore172.23.137.160 ERROR ====================================================================== ERROR: test_multiple_backups_merges (ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 1864, in validate_backup_data .format(bucket.name)) Exception: Data not match in backup bucket default During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/ent_backup_restore/enterprise_backup_merge_tests.py", line 39, in test_multiple_backups_merges self.do_backup_merge_actions() File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 4221, in do_backup_merge_actions self.backup_merge_actions[action](self) File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 3407, in merge self.backup_merge_validate(repeats=self.number_of_repeats, skip_validation=skip_validation) File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 1718, in backup_merge_validate self.validate_backup_data(**kwargs) File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 1872, in validate_backup_data raise Exception("\nMissing key: {0}".format(key)) Exception: Missing key: ent-backup32 ---------------------------------------------------------------------- Ran 1 test in 1488.840s FAILED (errors=1) summary so far suite ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest , pass 0 , fail 1