2021-07-13 10:23:53,930 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:23:55,073 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:23:55,808 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:322] Initializing cluster 2021-07-13 10:24:02,098 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:04,532 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:06,552 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:08,572 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:10,589 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:12,608 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:14,630 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:16,648 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:18,667 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:27,638 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:29,657 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:31,673 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:33,688 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:35,706 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:37,720 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:39,733 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:41,749 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:46,796 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:48,812 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:50,826 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:24:52,842 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:02,213 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:04,226 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:06,240 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:08,256 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:10,273 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:12,288 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:14,305 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:16,319 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:22,234 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:24,249 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:26,263 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:28,319 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-07-13 10:25:44,658 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:599] RAM quota was defined less than 100 MB: 2021-07-13 10:25:58,855 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:344] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.120.206, 172.23.104.186, 172.23.120.201 initialized 2021-07-13 10:26:04,651 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:26:04,674 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= ClusterSetup setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:26:23,834 | test | INFO | MainProcess | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.1.0-1066-enterprise | 0.629722921914 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.201 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-07-13 10:26:23,851 | test | INFO | MainProcess | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-07-13 10:26:33,910 | test | INFO | MainProcess | pool-3-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-07-13 10:26:33,930 | test | INFO | MainProcess | pool-3-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.0980000496 sec 2021-07-13 10:26:34,164 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 10.0640204866 | 3949948928 | 3094196224 | 58195968 / 3909087232 | 533800 / 1046278 | 7.1.0-1066-enterprise | | 172.23.98.195 | kv | 4.94076128056 | 3949948928 | 3248484352 | 82837504 / 3909087232 | 0 / 0 | 7.1.0-1066-enterprise | | 172.23.104.186 | kv | 6.00600600601 | 6078930944 | 5365755904 | 229789696 / 3758092288 | 0 / 0 | 7.1.0-1066-enterprise | | 172.23.120.201 | kv | 2.50563768479 | 3945754624 | 3333566464 | 3670016 / 3758092288 | 0 / 0 | 7.1.0-1066-enterprise | | 172.23.120.206 | kv | 2.18483174284 | 3945754624 | 3330277376 | 0 / 0 | 0 / 0 | 7.1.0-1066-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-07-13 10:26:34,164 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= ClusterSetup setup complete for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:26:34,165 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= CollectionBase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:26:34,473 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1878] Creating required buckets from template 2021-07-13 10:26:53,045 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2343] Waiting for all collections to be created 2021-07-13 10:26:53,164 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-------------+-----------+-----------+ | fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000 | couchbase | magma | 2 | none | 0 | 0 | 10485760000 | 526989168 | 285144176 | +-----------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-07-13 10:26:53,165 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:158] Creating required SDK clients for client_pool 2021-07-13 10:27:04,430 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4679] Performing scope/collection specific operations 2021-07-13 10:27:04,444 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4769] Done Performing scope/collection specific operations 2021-07-13 10:30:16,653 | test | INFO | MainProcess | pool-3-thread-9 | [table_view:display:72] Ops trend for bucket 'fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000' +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................*..........................**....................***...........................*.......X | 30376.200 | | 29089.200 | ..............................................................................................**X | 29089.200 | | 26410.900 | .....................................................................................X | 26410.900 | | 22076.900 | .......................................................................*X | 22076.900 | | 21820.300 | ......................................................................**....*.*X | 23109.200 | | 22129.800 | .......................................................................*X | 22129.800 | | 21497.500 | .....................................................................X | 21497.500 | | 21019.100 | ....................................................................X | 21019.100 | | 19965.300 | ................................................................*...**X | 20686.500 | | 19880.900 | ................................................................X | 19880.900 | | 18970.400 | .............................................................*X | 18970.400 | | 18180.800 | ..........................................................*X | 18180.800 | | 17156.900 | .......................................................X | 17156.900 | | 16485.500 | .....................................................X | 16485.500 | | 15182.900 | ................................................*X | 15182.900 | | 14246.700 | .............................................*X | 14246.700 | | 13323.400 | ..........................................X | 13323.400 | | 13045.500 | .........................................**.***....X | 14381.500 | | 13506.700 | ...........................................X | 13506.700 | | 13089.000 | ..........................................X | 13089.000 | | 12476.600 | ........................................***..*.......*.....***.X | 17272.400 | | 15064.000 | ................................................*X | 15064.000 | | 13166.600 | ..........................................**X | 13166.600 | | 10798.500 | ..................................X | 10798.500 | | 9050.200 | ............................**.....**..*.*....X | 12701.100 | | 12239.400 | .......................................**...*............*......*....*X | 20034.100 | | 19715.600 | ...............................................................X | 19715.600 | | 18390.500 | ...........................................................X | 18390.500 | | 17394.600 | ........................................................X | 17394.600 | | 15061.400 | ................................................*X | 15061.400 | | 14715.800 | ...............................................X | 14715.800 | | 14481.000 | ..............................................X | 14481.000 | | 12002.000 | ......................................X | 12002.000 | | 11661.400 | .....................................**X | 11661.400 | | 10267.400 | ................................X | 10267.400 | | 7751.600 | ........................X | 7751.600 | | 7533.200 | .......................X | 7533.200 | | 5665.100 | .................***..*..*...*........**...*...*X | 12072.000 | | 10345.800 | .................................X | 10345.800 | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ 2021-07-13 10:30:18,214 | test | INFO | MainProcess | pool-3-thread-17 | [table_view:display:72] Ops trend for bucket 'fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 10345.800 | ...................................................................................................X | 10345.800 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2021-07-13 10:30:18,315 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 69.5923136848 | 3949948928 | 2282582016 | 58195968 / 3909087232 | 1124825 / 2243109 | 7.1.0-1066-enterprise | | 172.23.98.195 | kv | 80.0673052032 | 3949948928 | 2446073856 | 82837504 / 3909087232 | 595311 / 1199843 | 7.1.0-1066-enterprise | | 172.23.104.186 | kv | 62.2912920627 | 6078930944 | 4557295616 | 229789696 / 3758092288 | 593739 / 1169036 | 7.1.0-1066-enterprise | | 172.23.120.201 | kv | 26.630988787 | 3945754624 | 2466050048 | 3670016 / 3758092288 | 578917 / 1143585 | 7.1.0-1066-enterprise | | 172.23.120.206 | kv | 25.5683269476 | 3945754624 | 2534592512 | 0 / 0 | 526374 / 1048205 | 7.1.0-1066-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-07-13 10:30:39,601 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4433] Validating collection stats and item counts 2021-07-13 10:31:10,937 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-------------+------------+------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-------------+------------+------------+ | fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000 | couchbase | magma | 2 | none | 0 | 3000000 | 10485760000 | 2396321616 | 2033717678 | +-----------------------------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-------------+------------+------------+ 2021-07-13 10:31:10,938 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:547] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-07-13 10:31:17,792 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:893] Doing collection data load during graceful_failover_recovery 2021-07-13 10:31:17,793 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:355] Starting rebalance operation of type : graceful_failover_recovery 2021-07-13 10:31:17,793 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:696] failing over nodes [ip:172.23.104.186 port:8091 ssh_username:root] 2021-07-13 10:31:39,167 | test | INFO | MainProcess | pool-3-thread-4 | [rest_client:monitorRebalance:1483] Rebalance done. Taken 21.2039999962 seconds to complete 2021-07-13 10:31:39,171 | test | INFO | MainProcess | pool-3-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-07-13 10:33:44,174 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:331] Starting a sync data load after failover 2021-07-13 10:33:44,177 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4679] Performing scope/collection specific operations 2021-07-13 10:33:44,184 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4769] Done Performing scope/collection specific operations 2021-07-13 10:34:43,085 | test | INFO | MainProcess | pool-3-thread-25 | [table_view:display:72] Ops trend for bucket 'fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000' +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **............*...................................**............................**.............*......*......X | 53985.200 | | 50154.800 | ...........................................................................................*X | 50154.800 | | 44978.500 | ..................................................................................X | 44978.500 | | 40508.800 | ..........................................................................X | 40508.800 | | 38163.700 | .....................................................................X | 38163.700 | | 35790.200 | .................................................................**X | 35790.200 | | 33981.200 | .............................................................X | 33981.200 | | 28481.600 | ...................................................X | 28481.600 | | 27584.600 | ..................................................**X | 27584.600 | | 26128.900 | ...............................................X | 26128.900 | | 25211.200 | .............................................X | 25211.200 | | 23039.900 | .........................................*...**....*...X | 28518.200 | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-07-13 10:34:47,250 | test | INFO | MainProcess | pool-3-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1007, in main() File "/opt/couchbase/lib/python/cbstats", line 1004, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-07-13 10:34:52,546 | test | INFO | MainProcess | pool-3-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1007, in main() File "/opt/couchbase/lib/python/cbstats", line 1004, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-07-13 10:34:57,822 | test | INFO | MainProcess | pool-3-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1007, in main() File "/opt/couchbase/lib/python/cbstats", line 1004, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-07-13 10:35:03,098 | test | INFO | MainProcess | pool-3-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1007, in main() File "/opt/couchbase/lib/python/cbstats", line 1004, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-07-13 10:35:08,382 | test | INFO | MainProcess | pool-3-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1007, in main() File "/opt/couchbase/lib/python/cbstats", line 1004, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-07-13 10:35:22,252 | test | INFO | MainProcess | pool-3-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.1.0-1066-enterprise | 9.23154958154 | Cluster node | | 172.23.98.195 | kv | 7.1.0-1066-enterprise | 9.56477732794 | Cluster node | | 172.23.104.186 | kv | 7.1.0-1066-enterprise | 2.95665246805 | Cluster node | | 172.23.120.201 | kv | 7.1.0-1066-enterprise | 5.35310379492 | Cluster node | | 172.23.120.206 | kv | 7.1.0-1066-enterprise | 29.4994944388 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-07-13 10:35:22,263 | test | INFO | MainProcess | pool-3-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-07-13 10:35:27,351 | test | INFO | MainProcess | pool-3-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-07-13 10:35:32,391 | test | INFO | MainProcess | pool-3-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-07-13 10:35:37,430 | test | INFO | MainProcess | pool-3-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-07-13 10:35:42,477 | test | INFO | MainProcess | pool-3-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-07-13 10:35:47,519 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1547] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'99b5c2239559044048ae81a597e1b405', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=0c8d2fefae24bf027e1eb01803b45984', u'status': u'notRunning'} - rebalance failed 2021-07-13 10:35:47,596 | test | INFO | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2688] Latest logs from UI on 172.23.98.196: 2021-07-13 10:35:47,598 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197747058L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:35:47.058Z', u'text': u'Rebalance exited with reason {badarg,\n [{io_lib,format,\n ["Rebalancing bucket ~p with config ~p",\n ["fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000",\n [{auth_type,sasl},\n {num_replicas,2},\n {replica_index,true},\n {ram_quota,2097152000},\n {durability_min_level,none},\n {frag_percent,50},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {autocompaction,false},\n {purge_interval,undefined},\n {flush_enabled,false},\n {num_threads,3},\n {eviction_policy,full_eviction},\n {conflict_resolution_type,seqno},\n {storage_mode,magma},\n {max_ttl,0},\n {compression_mode,active},\n {type,membase},\n {num_vbuckets,1024},\n {replication_topology,star},\n {repl_type,dcp},\n {servers,\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\']},\n {map_opts_hash,133465355},\n {map,\n [[\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.201\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.120.206\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.195\',\n \'ns_1@172.23.98.196\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',undefined],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.120.206\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\',\n \'ns_1@172.23.98.195\'],\n [\'ns_1@172.23.120.201\',\n \'ns_1@172.23.98.196\'|...],\n [\'ns_1@172.23.120.201\'|...],\n [...]|...]},\n {fastForwardMap,undefined}]]],\n [{file,"io_lib.erl"},{line,202}]},\n {\'ale_logger-rebalance\',info,5,\n [{file,[]},{line,39}]},\n {ns_rebalancer,rebalance_bucket,6,\n [{file,"src/ns_rebalancer.erl"},{line,622}]},\n {lists,foreach,2,\n [{file,"lists.erl"},{line,1338}]},\n {ns_rebalancer,rebalance_kv,4,\n [{file,"src/ns_rebalancer.erl"},{line,605}]},\n {ns_rebalancer,rebalance_body,5,\n [{file,"src/ns_rebalancer.erl"},{line,556}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{...}]}]}.\nRebalance Operation Id = ...'} 2021-07-13 10:35:47,607 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197746917L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:35:46.917Z', u'text': u'Started rebalancing bucket fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000'} 2021-07-13 10:35:47,607 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1626197744575L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:35:44.575Z', u'text': u'Bucket "fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000" loaded on node \'ns_1@172.23.104.186\' in 10 seconds.'} 2021-07-13 10:35:47,607 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197722186L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:35:22.186Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.98.196','ns_1@172.23.98.195',\n 'ns_1@172.23.104.186','ns_1@172.23.120.201',\n 'ns_1@172.23.120.206'], EjectNodes = [], Failed over and being ejected nodes = [], Delta recovery nodes = ['ns_1@172.23.104.186'], Delta recovery buckets = all; Operation Id = d319816316597fb3481a224f7b0ca331"} 2021-07-13 10:35:47,608 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197497543L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:31:37.543Z', u'text': u'Graceful failover completed successfully.\nRebalance Operation Id = 77e187f31664d41b30c2d7d888585d2a'} 2021-07-13 10:35:47,608 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1626197497518L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:31:37.518Z', u'text': u'Shutting down bucket "fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000" on \'ns_1@172.23.104.186\' for deletion'} 2021-07-13 10:35:47,608 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197497354L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:31:37.354Z', u'text': u"Deactivating failed over nodes ['ns_1@172.23.104.186']"} 2021-07-13 10:35:47,611 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197497333L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:31:37.333Z', u'text': u"Failed over ['ns_1@172.23.104.186']: ok"} 2021-07-13 10:35:47,611 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197496999L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:31:36.999Z', u'text': u"Starting failing over ['ns_1@172.23.104.186']"} 2021-07-13 10:35:47,612 | test | ERROR | MainProcess | pool-3-thread-22 | [rest_client:print_UI_logs:2690] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1626197478175L, u'shortText': u'message', u'serverTime': u'2021-07-13T10:31:18.175Z', u'text': u'Bucket "fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000" rebalance does not seem to be swap rebalance'} 2021-07-13 10:35:47,612 | test | ERROR | MainProcess | pool-3-thread-22 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'99b5c2239559044048ae81a597e1b405', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=0c8d2fefae24bf027e1eb01803b45984', u'status': u'notRunning'} - rebalance failed 2021-07-13 10:35:49,229 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:79] Bucket: fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000, Active Resident ratio(DGM): 100% 2021-07-13 10:35:49,229 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: fW7UdP9XFESEaeyN9jyZh-P95aEjty7iOj4IOQSKlHTT0RORo4%og42zG37TitH7%-NP_51-34-501000, Replica Resident ratio(DGM): 100% 2021-07-13 10:36:32,410 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1095] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.120.201,ns_1@172.23.120.206,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-07-13 10:36:42,443 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1098] ns_1@172.23.104.186,ns_1@172.23.120.201,ns_1@172.23.120.206,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-07-13 10:36:42,444 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1103] Polling active_tasks to check cbcollect status 2021-07-13 10:44:23,434 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1130] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-07-13 10:44:25,401 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1130] ns_1@172.23.120.201: Copying cbcollect ZIP file to Client 2021-07-13 10:44:26,743 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1130] ns_1@172.23.120.206: Copying cbcollect ZIP file to Client 2021-07-13 10:44:33,688 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1130] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-07-13 10:44:35,710 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1130] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-07-13 10:44:37,944 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:494] Skipping get_trace !! 2021-07-13 10:45:16,049 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1483] Rebalance done. Taken 11.0770001411 seconds to complete 2021-07-13 10:45:16,059 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete