2023-09-14 20:35:52,950 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ConcurrentFailoverTests setup started for test #1 test_concurrent_failover ========= 2023-09-14 20:35:56,944 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= OnPremBaseTest setup started for test #1 test_concurrent_failover ========= 2023-09-14 20:35:56,946 | test | INFO | MainProcess | MainThread | [onPrem_basetestcase:setUp:245] Delete all buckets and rebalance out other nodes from 'C1' 2023-09-14 20:36:07,483 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1312] Rebalance done. Taken 3.27800011635 seconds to complete 2023-09-14 20:36:07,483 | test | INFO | MainProcess | MainThread | [common_lib:sleep:20] Sleep 10 seconds. Reason: Wait after rebalance complete 2023-09-14 20:36:21,658 | test | INFO | MainProcess | MainThread | [onPrem_basetestcase:enable_tls_on_nodes:510] Validating if services obey tls only on servers [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.67 port:8091 ssh_username:root, ip:172.23.110.68 port:8091 ssh_username:root] 2023-09-14 20:36:22,364 | test | INFO | MainProcess | MainThread | [common_lib:sleep:20] Sleep 2 seconds. Reason: waiting after enabling TLS 2023-09-14 20:36:24,627 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= OnPremBaseTest setup finished for test #1 test_concurrent_failover ========= 2023-09-14 20:36:24,861 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= OnPremBaseTest setup finished for test #1 test_concurrent_failover ========= 2023-09-14 20:36:24,862 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ClusterSetup setup started for test #1 test_concurrent_failover ========= 2023-09-14 20:36:54,898 | test | CRITICAL | MainProcess | pool-5-thread-17 | [cluster_ready_functions:validate_orchestrator_selection:292] Orchestrator: 172.23.110.64 2023-09-14 20:36:55,219 | test | INFO | MainProcess | pool-5-thread-17 | [table_view:display:72] Rebalance Overview +---------------+---------+----------+---------------------------------+----------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------+---------------------------------+----------------+--------------+-----------------------+ | 172.23.110.64 | Group 1 | kv | 7.6.0-1440-enterprise / default | 0.288986332058 | Cluster node | active / none | | 172.23.110.65 | None | kv | | | <--- IN --- | | | 172.23.110.66 | None | kv | | | <--- IN --- | | | 172.23.110.67 | None | kv | | | <--- IN --- | | | 172.23.110.68 | None | kv | | | <--- IN --- | | +---------------+---------+----------+---------------------------------+----------------+--------------+-----------------------+ 2023-09-14 20:37:00,565 | test | INFO | MainProcess | pool-5-thread-17 | [task:check:832] Rebalance - status: none, progress: 100 2023-09-14 20:37:00,693 | test | INFO | MainProcess | pool-5-thread-17 | [task:check:891] Rebalance completed with progress: 100% in 5.47399997711 sec 2023-09-14 20:37:01,170 | test | CRITICAL | MainProcess | pool-5-thread-17 | [cluster_ready_functions:validate_orchestrator_selection:292] Orchestrator: 172.23.110.64 2023-09-14 20:37:01,171 | infra | CRITICAL | MainProcess | pool-5-thread-17 | [task:print_nodes:622] Cluster nodes..: ['172.23.110.64:18091', '172.23.110.66:18091', '172.23.110.67:18091', '172.23.110.68:18091', '172.23.110.65:18091'] 2023-09-14 20:37:01,173 | infra | CRITICAL | MainProcess | pool-5-thread-17 | [task:print_nodes:622] KV............: ['172.23.110.64:18091', '172.23.110.66:18091', '172.23.110.67:18091', '172.23.110.68:18091', '172.23.110.65:18091'] 2023-09-14 20:37:02,553 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | 172.23.110.67 | Group 1 | kv | 0.0918 | 251.34 GiB | 246.22 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.64 | Group 1 | kv | 0.2579 | 251.34 GiB | 246.13 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.68 | Group 1 | kv | 0.0912 | 251.34 GiB | 247.06 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.66 | Group 1 | kv | 0.2263 | 251.34 GiB | 248.04 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.65 | Group 1 | kv | 0.2029 | 251.34 GiB | 246.74 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ 2023-09-14 20:37:02,556 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ClusterSetup setup complete for test #1 test_concurrent_failover ========= 2023-09-14 20:37:02,696 | test | INFO | MainProcess | MainThread | [collections_base:deploy_buckets_from_spec_file:310] Creating bucket from spec: single_bucket.default 2023-09-14 20:37:02,697 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:2542] Creating required buckets from template 2023-09-14 20:37:04,357 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:3369] Waiting for all collections to be created 2023-09-14 20:37:04,844 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | 172.23.110.67 | Group 1 | kv | 0.0918 | 251.34 GiB | 246.22 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.64 | Group 1 | kv | 0.2579 | 251.34 GiB | 246.13 GiB | 0.0 Byte / 4.00 GiB | 41050 / 120860 | 7.6.0-1440-enterprise / default | | 172.23.110.68 | Group 1 | kv | 0.0912 | 251.34 GiB | 247.00 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.66 | Group 1 | kv | 0.2263 | 251.34 GiB | 248.04 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.65 | Group 1 | kv | 0.2029 | 251.34 GiB | 246.74 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ 2023-09-14 20:37:05,431 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+------------+-----+ | Bucket | Type | Storage | Replicas | Durability | TTL | Items | Vbuckets | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+------------+-----+ | bucket-0 | couchbase | magma | 3 | none | 0 | 41050 | 1024 | 673.58 GiB | 179.39 MiB | 167.04 MiB | 100 | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+------------+-----+ 2023-09-14 20:37:05,433 | test | INFO | MainProcess | MainThread | [collections_base:create_clients_for_sdk_pool:339] Creating required SDK clients for client_pool 2023-09-14 20:37:36,095 | test | INFO | MainProcess | MainThread | [collections_base:load_data_from_spec_file:349] Load docs using spec file initial_load 2023-09-14 20:37:36,108 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5988] Performing scope/collection specific operations 2023-09-14 20:37:36,118 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:6095] Done Performing scope/collection specific operations 2023-09-14 20:37:37,773 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | 172.23.110.67 | Group 1 | kv | 0.2925 | 251.34 GiB | 246.11 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.64 | Group 1 | kv | 0.5238 | 251.34 GiB | 246.01 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.68 | Group 1 | kv | 0.2761 | 251.34 GiB | 246.91 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.66 | Group 1 | kv | 0.2906 | 251.34 GiB | 247.91 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.65 | Group 1 | kv | 0.2195 | 251.34 GiB | 246.62 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ 2023-09-14 20:37:37,775 | test | INFO | MainProcess | MainThread | [collections_base:load_data_from_spec_file:378] Wait for ep_queue_size to drain 2023-09-14 20:37:51,575 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:5573] Validating collection stats and item counts 2023-09-14 20:38:09,358 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | 172.23.110.67 | Group 1 | kv | 0.2157 | 251.34 GiB | 246.11 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.64 | Group 1 | kv | 0.3025 | 251.34 GiB | 246.02 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.68 | Group 1 | kv | 0.2530 | 251.34 GiB | 246.90 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.66 | Group 1 | kv | 0.2594 | 251.34 GiB | 247.88 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.65 | Group 1 | kv | 0.1600 | 251.34 GiB | 246.61 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ 2023-09-14 20:38:09,993 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | Bucket | Type | Storage | Replicas | Durability | TTL | Items | Vbuckets | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | bucket-0 | couchbase | magma | 3 | none | 0 | 0 | 1024 | 673.58 GiB | 510.07 MiB | 27.96 MiB | 100 | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ 2023-09-14 20:38:11,298 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ | 172.23.110.67 | Group 1 | kv | 0.2749 | 251.34 GiB | 246.11 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.64 | Group 1 | kv | 0.3025 | 251.34 GiB | 246.02 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.68 | Group 1 | kv | 0.2530 | 251.34 GiB | 246.90 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.66 | Group 1 | kv | 0.2594 | 251.34 GiB | 247.88 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | | 172.23.110.65 | Group 1 | kv | 0.1600 | 251.34 GiB | 246.61 GiB | 0.0 Byte / 4.00 GiB | 0 / 0 | 7.6.0-1440-enterprise / default | +---------------+---------+----------+--------+------------+------------+---------------------+-------------------+---------------------------------+ 2023-09-14 20:38:11,934 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | Bucket | Type | Storage | Replicas | Durability | TTL | Items | Vbuckets | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | bucket-0 | couchbase | magma | 3 | none | 0 | 0 | 1024 | 673.58 GiB | 510.07 MiB | 27.96 MiB | 100 | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ 2023-09-14 20:38:11,937 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ConcurrentFailoverTests setUp started for test #1 test_concurrent_failover ========= 2023-09-14 20:38:11,937 | test | INFO | MainProcess | MainThread | [concurrent_failovers:setUp:61] Updating Auto-failover settings 2023-09-14 20:38:12,155 | test | CRITICAL | MainProcess | MainThread | [table_view:display:72] Auto failover status: +---------+---------------+-----------------------+-----------------+--------------+----------------------+ | Enabled | Auto FO count | Max Events configured | Auto FO timeout | Disk Auto FO | Disk Auto FO timeout | +---------+---------------+-----------------------+-----------------+--------------+----------------------+ | True | 0 | 1 | 1 | False | 120 | +---------+---------------+-----------------------+-----------------+--------------+----------------------+ 2023-09-14 20:38:12,157 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ConcurrentFailoverTests setUp complete for test #1 test_concurrent_failover ========= 2023-09-14 20:38:12,780 | test | INFO | MainProcess | MainThread | [concurrent_failovers:_ConcurrentFailoverTests__update_unaffected_node:387] Node for REST APIs: 172.23.110.64 2023-09-14 20:38:12,891 | test | CRITICAL | MainProcess | MainThread | [table_view:display:72] Auto failover status: +---------+---------------+-----------------------+-----------------+--------------+----------------------+ | Enabled | Auto FO count | Max Events configured | Auto FO timeout | Disk Auto FO | Disk Auto FO timeout | +---------+---------------+-----------------------+-----------------+--------------+----------------------+ | True | 0 | 1 | 1 | False | 120 | +---------+---------------+-----------------------+-----------------+--------------+----------------------+ 2023-09-14 20:38:13,006 | test | CRITICAL | MainProcess | MainThread | [table_view:display:72] Nodes to be failed +---------------+----------+-------------+----------------+ | Node | Services | Node status | Failover type | +---------------+----------+-------------+----------------+ | 172.23.110.65 | kv | active | stop_memcached | +---------------+----------+-------------+----------------+ 2023-09-14 20:38:13,234 | test | INFO | MainProcess | MainThread | [concurrent_failovers:num_nodes_to_be_failover:274] Expected nodes to be failed over: 1 2023-09-14 20:38:30,045 | test | CRITICAL | MainProcess | pool-5-thread-28 | [task:call:6126] Failover not started as expected 2023-09-14 20:38:30,046 | test | CRITICAL | MainProcess | pool-5-thread-28 | [task:call:6130] Auto-Failover rebalance failed 2023-09-14 20:38:30,046 | test | ERROR | MainProcess | MainThread | [concurrent_failovers:_ConcurrentFailoverTests__run_test:462] Exception occurred: Failure during concurrent failover procedure 2023-09-14 20:38:30,861 | test | INFO | MainProcess | MainThread | [common_lib:sleep:20] Sleep 15 seconds. Reason: Wait before enabling back auto-fo 2023-09-14 20:38:46,115 | test | CRITICAL | MainProcess | MainThread | [table_view:display:72] Nodes status failure +---------------+----------+-------------+----------------+ | Node | Services | Node status | Failover type | +---------------+----------+-------------+----------------+ | 172.23.110.65 | kv | active | stop_memcached | +---------------+----------+-------------+----------------+ 2023-09-14 20:38:46,776 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | Bucket | Type | Storage | Replicas | Durability | TTL | Items | Vbuckets | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | bucket-0 | couchbase | magma | 3 | none | 0 | 0 | 1024 | 673.58 GiB | 510.14 MiB | 27.96 MiB | 100 | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ 2023-09-14 20:38:46,894 | test | CRITICAL | MainProcess | MainThread | [table_view:display:72] Auto failover status: +---------+---------------+-----------------------+-----------------+--------------+----------------------+ | Enabled | Auto FO count | Max Events configured | Auto FO timeout | Disk Auto FO | Disk Auto FO timeout | +---------+---------------+-----------------------+-----------------+--------------+----------------------+ | True | 0 | 1 | 1 | False | 120 | +---------+---------------+-----------------------+-----------------+--------------+----------------------+ 2023-09-14 20:38:53,690 | test | ERROR | MainProcess | MainThread | [rest_client:_rebalance_status_and_progress:1379] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'2e289d3b34c9b1615a36412623069282', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4265327ee10600f362a976fa49a10a6b', u'status': u'notRunning'} - rebalance failed 2023-09-14 20:38:53,844 | test | INFO | MainProcess | MainThread | [rest_client:print_UI_logs:2575] Latest logs from UI on 172.23.110.64: 2023-09-14 20:38:53,845 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1694749131264L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:38:51.264Z', u'text': u'Rebalance exited with reason {{badmatch,\n {error,\n {failed_nodes,[\'ns_1@172.23.110.64\']}}},\n [{ns_janitor,cleanup_apply_config_body,4,\n [{file,"src/ns_janitor.erl"},{line,428}]},\n {ns_janitor,\n \'-cleanup_apply_config_on_buckets/3-fun-1-\',\n 3,\n [{file,"src/ns_janitor.erl"},{line,383}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,199}]}]}.\nRebalance Operation Id = 29257da2a405f2dc4c95d4e3c64895c5'} 2023-09-14 20:38:53,845 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1694749127298L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:38:47.298Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.67','ns_1@172.23.110.64',\n 'ns_1@172.23.110.68','ns_1@172.23.110.66',\n 'ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 29257da2a405f2dc4c95d4e3c64895c5"} 2023-09-14 20:38:53,846 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'auto_failover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1694749110165L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:38:30.165Z', u'text': u'Disabled auto-failover'} 2023-09-14 20:38:53,846 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'auto_failover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1694749092047L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:38:12.047Z', u'text': u'Enabled auto-failover with timeout 1 and max count 1'} 2023-09-14 20:38:53,848 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1694749082761L, u'shortText': u'node up', u'serverTime': u'2023-09-14T20:38:02.761Z', u'text': u"Node 'ns_1@172.23.110.64' saw that node 'ns_1@172.23.110.68' came up. Tags: [] (repeated 1 times, last seen 72.045921 secs ago)"} 2023-09-14 20:38:53,848 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1694749058255L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:37:38.255Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>] (repeated 1 times, last seen 51.247939 secs ago)'} 2023-09-14 20:38:53,848 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1694749051111L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:37:31.111Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>] (repeated 2 times, last seen 44.124375 secs ago)'} 2023-09-14 20:38:53,848 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1694749024169L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:37:04.169Z', u'text': u'Bucket "bucket-0" loaded on node \'ns_1@172.23.110.68\' in 0 seconds.'} 2023-09-14 20:38:53,849 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.67', u'tstamp': 1694749024164L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:37:04.164Z', u'text': u'Bucket "bucket-0" loaded on node \'ns_1@172.23.110.67\' in 0 seconds.'} 2023-09-14 20:38:53,849 | test | ERROR | MainProcess | MainThread | [rest_client:print_UI_logs:2577] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1694749024160L, u'shortText': u'message', u'serverTime': u'2023-09-14T20:37:04.160Z', u'text': u'Bucket "bucket-0" loaded on node \'ns_1@172.23.110.66\' in 0 seconds.'} 2023-09-14 20:38:53,859 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ConcurrentFailoverTests tearDown started for test #1 test_concurrent_failover ========= 2023-09-14 20:38:54,111 | test | INFO | MainProcess | MainThread | [concurrent_failovers:tearDown:103] Resetting auto-failover settings to default 2023-09-14 20:38:54,223 | test | INFO | MainProcess | MainThread | [cb_basetest:log_setup_status:212] ========= ConcurrentFailoverTests tearDown complete for test #1 test_concurrent_failover ========= 2023-09-14 20:38:54,223 | test | INFO | MainProcess | MainThread | [AutoFailoverBaseTest:tearDown:145] ============AutoFailoverBaseTest teardown============ 2023-09-14 20:38:54,841 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | Bucket | Type | Storage | Replicas | Durability | TTL | Items | Vbuckets | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ | bucket-0 | couchbase | magma | 3 | none | 0 | 0 | 1024 | 673.58 GiB | 510.14 MiB | 27.96 MiB | 100 | +----------+-----------+---------+----------+------------+-----+-------+----------+------------+------------+-----------+-----+ 2023-09-14 20:39:25,272 | test | CRITICAL | MainProcess | MainThread | [onPrem_basetestcase:tearDownEverything:625] Skipping get_trace !!