2021-11-12 01:30:02,506 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:30:03,226 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:30:03,493 | test | INFO | MainProcess | MainThread | [basetestcase:initialize_cluster:404] Initializing cluster : C1 2021-11-12 01:30:21,970 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.101:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:30:23,983 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.101:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:30:25,994 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.101:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:30:28,007 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.101:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:30:30,019 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.101:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:30:32,032 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.101:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:00,951 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.104:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:02,967 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.104:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:04,980 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.104:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:06,993 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.104:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:09,006 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.104:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:11,019 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.104:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:33,513 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.103:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:35,525 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.103:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:37,539 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.103:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:39,552 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.103:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:41,565 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.103:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:31:43,578 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.103:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:05,296 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.102:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:07,309 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.102:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:09,321 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.102:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:11,332 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.102:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:13,345 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.102:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:15,358 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.102:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:36,835 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.106:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:38,845 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.106:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:40,857 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.106:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:42,871 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.106:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:32:44,882 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.106:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:06,198 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.105:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:08,211 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.105:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:10,226 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.105:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:12,239 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.105:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:14,250 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.105:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:16,265 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.136.105:8091/nodes/self error [Errno 111] Connection refused 2021-11-12 01:33:35,128 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] POST http://172.23.136.101:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["An absolute path is required for path","An absolute path is required for index_path"] auth: Administrator:password 2021-11-12 01:33:35,132 | test | ERROR | MainProcess | MainThread | [rest_client:set_data_path:2232] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["An absolute path is required for path","An absolute path is required for index_path"] 2021-11-12 01:33:35,279 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] POST http://172.23.136.104:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["An absolute path is required for path","An absolute path is required for index_path"] auth: Administrator:password 2021-11-12 01:33:35,282 | test | ERROR | MainProcess | MainThread | [rest_client:set_data_path:2232] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["An absolute path is required for path","An absolute path is required for index_path"] 2021-11-12 01:33:35,433 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] POST http://172.23.136.103:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["An absolute path is required for path","An absolute path is required for index_path"] auth: Administrator:password 2021-11-12 01:33:35,434 | test | ERROR | MainProcess | MainThread | [rest_client:set_data_path:2232] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["An absolute path is required for path","An absolute path is required for index_path"] 2021-11-12 01:33:35,586 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] POST http://172.23.136.102:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["An absolute path is required for path","An absolute path is required for index_path"] auth: Administrator:password 2021-11-12 01:33:35,589 | test | ERROR | MainProcess | MainThread | [rest_client:set_data_path:2232] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["An absolute path is required for path","An absolute path is required for index_path"] 2021-11-12 01:33:35,703 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] POST http://172.23.136.106:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["An absolute path is required for path","An absolute path is required for index_path"] auth: Administrator:password 2021-11-12 01:33:35,706 | test | ERROR | MainProcess | MainThread | [rest_client:set_data_path:2232] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["An absolute path is required for path","An absolute path is required for index_path"] 2021-11-12 01:33:35,848 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:822] POST http://172.23.136.105:8091//nodes/self/controller/settings body: path=%2Fdata&index_path=%2Fdata headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["An absolute path is required for path","An absolute path is required for index_path"] auth: Administrator:password 2021-11-12 01:33:35,851 | test | ERROR | MainProcess | MainThread | [rest_client:set_data_path:2232] Unable to set data_path path=%2Fdata&index_path=%2Fdata: ["An absolute path is required for path","An absolute path is required for index_path"] 2021-11-12 01:33:36,038 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:700] RAM quota was defined less than 100 MB: 2021-11-12 01:33:50,700 | test | INFO | MainProcess | MainThread | [basetestcase:initialize_cluster:429] Cluster C1 initialized 2021-11-12 01:33:50,701 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:33:50,723 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= ClusterSetup setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:34:10,490 | test | INFO | MainProcess | pool-6-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.136.101 | kv | 7.1.0-1696-enterprise | 6.14333333333 | Cluster node | | 172.23.136.104 | None | | | <--- IN --- | | 172.23.136.103 | None | | | <--- IN --- | | 172.23.136.102 | None | | | <--- IN --- | | 172.23.136.106 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-11-12 01:34:10,503 | test | INFO | MainProcess | pool-6-thread-7 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-11-12 01:34:25,549 | test | INFO | MainProcess | pool-6-thread-7 | [task:check:384] Rebalance - status: none, progress: 100 2021-11-12 01:34:25,565 | test | INFO | MainProcess | pool-6-thread-7 | [task:check:443] Rebalance completed with progress: 100% in 15.0750000477 sec 2021-11-12 01:34:25,729 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+---------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+---------------------+-------------------+-----------------------+ | 172.23.136.106 | kv | 0.156666666667 | 6.00 GiB | 4.58 GiB | 1.63 GiB / 7.00 GiB | 0 / 0 | 7.1.0-1696-enterprise | | 172.23.136.103 | kv | 0.991553433713 | 6.00 GiB | 4.52 GiB | 1.68 GiB / 7.00 GiB | 0 / 0 | 7.1.0-1696-enterprise | | 172.23.136.104 | kv | 0.91475119769 | 6.00 GiB | 4.54 GiB | 1.67 GiB / 7.00 GiB | 0 / 0 | 7.1.0-1696-enterprise | | 172.23.136.101 | kv | 0.910015166919 | 6.00 GiB | 4.48 GiB | 1.72 GiB / 7.00 GiB | 713544 / 1420352 | 7.1.0-1696-enterprise | | 172.23.136.102 | kv | 3.10156328122 | 6.00 GiB | 4.55 GiB | 1.63 GiB / 7.00 GiB | 0 / 0 | 7.1.0-1696-enterprise | +----------------+----------+-----------------+-----------+----------+---------------------+-------------------+-----------------------+ 2021-11-12 01:34:25,730 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= ClusterSetup setup complete for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:34:25,730 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= CollectionBase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:34:25,989 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2021-11-12 01:34:43,112 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2484] Waiting for all collections to be created 2021-11-12 01:34:44,005 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000 | couchbase | magma | 2 | none | 0 | 0 | 9.77 GiB | 725.49 MiB | 94.99 MiB | 100 | +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ 2021-11-12 01:34:44,006 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2021-11-12 01:34:54,594 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4861] Performing scope/collection specific operations 2021-11-12 01:34:54,599 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4951] Done Performing scope/collection specific operations 2021-11-12 01:37:47,690 | test | INFO | MainProcess | pool-6-thread-10 | [table_view:display:72] Ops trend for bucket 'L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000' +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***............................................................................*X | 15781.600 | | 15324.900 | .........................................................................***......**........**......**.***.***....*X | 20528.700 | | 18276.700 | ........................................................................................*X | 18276.700 | | 14679.400 | ......................................................................*..................X | 18285.600 | | 17112.300 | ..................................................................................*X | 17112.300 | | 16478.100 | ...............................................................................**..........*X | 18586.600 | | 17761.300 | .....................................................................................**X | 17761.300 | | 16963.833 | .................................................................................**.**X | 17090.525 | +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ 2021-11-12 01:37:50,186 | test | INFO | MainProcess | pool-6-thread-24 | [table_view:display:72] Ops trend for bucket 'L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 13299.800 | ...................................................................................................X | 13299.800 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2021-11-12 01:37:50,394 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+---------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+---------------------+-------------------+-----------------------+ | 172.23.136.106 | kv | 97.8100364994 | 6.00 GiB | 2.25 GiB | 3.82 GiB / 7.00 GiB | 595913 / 1207488 | 7.1.0-1696-enterprise | | 172.23.136.103 | kv | 99.7666992074 | 6.00 GiB | 2.47 GiB | 3.68 GiB / 7.00 GiB | 565428 / 1014069 | 7.1.0-1696-enterprise | | 172.23.136.104 | kv | 99.371395135 | 6.00 GiB | 2.45 GiB | 3.69 GiB / 7.00 GiB | 562331 / 1140735 | 7.1.0-1696-enterprise | | 172.23.136.101 | kv | 99.9241612398 | 6.00 GiB | 2.52 GiB | 3.63 GiB / 7.00 GiB | 1283668 / 2466525 | 7.1.0-1696-enterprise | | 172.23.136.102 | kv | 99.685687058 | 6.00 GiB | 2.51 GiB | 3.64 GiB / 7.00 GiB | 567855 / 1047142 | 7.1.0-1696-enterprise | +----------------+----------+-----------------+-----------+----------+---------------------+-------------------+-----------------------+ 2021-11-12 01:38:05,474 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4615] Validating collection stats and item counts 2021-11-12 01:38:40,154 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ | L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000 | couchbase | magma | 2 | none | 0 | 3000000 | 9.77 GiB | 6.15 GiB | 3.50 GiB | 100 | +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ 2021-11-12 01:38:40,154 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:648] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-11-12 01:38:46,733 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:896] Doing collection data load during swap_rebalance 2021-11-12 01:38:46,733 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:357] Starting rebalance operation of type : swap_rebalance 2021-11-12 01:38:52,091 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4861] Performing scope/collection specific operations 2021-11-12 01:38:52,098 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4951] Done Performing scope/collection specific operations 2021-11-12 01:38:52,213 | test | INFO | MainProcess | pool-6-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.136.106 | kv | 7.1.0-1696-enterprise | 10.4666666667 | --- OUT ---> | | 172.23.136.103 | kv | 7.1.0-1696-enterprise | 12.5886140348 | Cluster node | | 172.23.136.104 | kv | 7.1.0-1696-enterprise | 9.06166666667 | Cluster node | | 172.23.136.101 | kv | 7.1.0-1696-enterprise | 10.4714921418 | Cluster node | | 172.23.136.102 | kv | 7.1.0-1696-enterprise | 9.06166666667 | --- OUT ---> | | 172.23.136.105 | kv | 7.1.0-1696-enterprise | 0 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-11-12 01:38:52,223 | test | INFO | MainProcess | pool-6-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-11-12 01:39:02,285 | test | INFO | MainProcess | pool-6-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-11-12 01:39:12,348 | test | INFO | MainProcess | pool-6-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-11-12 01:39:25,506 | test | INFO | MainProcess | pool-6-thread-22 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-11-12 01:39:27,953 | test | ERROR | MainProcess | pool-6-thread-8 | [task:execute_tasks:3922] Failed to load 38 docs from 0 to 3000 of thread_name LoadDocs_MutateDocsFromSpecTask_1636709934.02_L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000_S%0nHgf%nmvR0z-26-19000_LsekIovHoHZLXZ8M4A2veGx5ThLb9uRP0oa0MKELGxI9iuV0fqhHtp7CIOpO434FkfgBgj9efSPBjJoezOzcyTZ66PzC6ZNWqUIfSvuWPudEmy24nyhbx2cqpKWmwyEMIGuav57UGpT6DNRn9dJDIFwcQeoq-mTJiYZiQ6f8kdJMY3ntbCo6QuvhF1hz_pqpfFq5s-PdeOY-26-21000_ttl=0_delete__0_3000_1636709934.28 2021-11-12 01:39:35,585 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:_rebalance_status_and_progress:1632] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fb98160c779f72d7dffbb5d62e873297', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=92c4ca28b447cbd9fb0b589f626bc5d7', u'status': u'notRunning'} - rebalance failed 2021-11-12 01:39:35,654 | test | INFO | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2782] Latest logs from UI on 172.23.136.101: 2021-11-12 01:39:35,654 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.101', u'tstamp': 1636709967895L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:39:27.895Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.1835.1>,\n {{{lost_connection,\n [{ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,209}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000\',\n {set_vbucket,994,active,\n [[\'ns_1@172.23.136.106\',\n \'ns_1@172.23.136.104\',\n \'ns_1@172.23.136.101\']]},\n 180000]}},\n {gen_server,call,\n [{\'janitor_agent-L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000\',\n \'ns_1@172.23.136.106\'},\n {if_rebalance,<0.511.1>,\n {update_vbucket_state,994,active,paused,\n undefined,\n [[\'ns_1@172.23.136.106\',\n \'ns_1@172.23.136.104\',\n \'ns_1@172.23.136.101\']]}},\n infinity]}}}}}.\nRebalance Operation Id = b413500a9e7ef72f63f40ab4f1e36723'} 2021-11-12 01:39:35,655 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.136.101', u'tstamp': 1636709967823L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:39:27.823Z', u'text': u'Worker <0.636.1> (for action {move,{994,\n [\'ns_1@172.23.136.106\',\n \'ns_1@172.23.136.104\',\n \'ns_1@172.23.136.101\'],\n [\'ns_1@172.23.136.104\',\n \'ns_1@172.23.136.101\',\n \'ns_1@172.23.136.103\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.1835.1>,\n {{{lost_connection,\n [{ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 209}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000\',\n {set_vbucket,\n 994,\n active,\n [[\'ns_1@172.23.136.106\',\n \'ns_1@172.23.136.104\',\n \'ns_1@172.23.136.101\']]},\n 180000]}},\n {gen_server,\n call,\n [{\'janitor_agent-L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000\',\n \'ns_1@172.23.136.106\'},\n {if_rebalance,\n <0.511.1>,\n {update_vbucket_state,\n 994,\n active,\n paused,\n undefined,\n [[\'ns_1@172.23.136.106\',\n \'ns_1@172.23.136.104\',\n \'ns_1@172.23.136.101\']]}},\n infinity]}}}}'} 2021-11-12 01:39:35,657 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.106', u'tstamp': 1636709967816L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:39:27.816Z', u'text': u"Control connection to memcached on 'ns_1@172.23.136.106' disconnected. Check logs for details."} 2021-11-12 01:39:35,657 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.136.106', u'tstamp': 1636709967700L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:39:27.700Z', u'text': u"Service 'memcached' exited with status -1073741819. Restarting. Messages:\nW1112 01:34:30.321171 4004 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\n2021-11-12T01:39:27.297618-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1696). Writing crash dump to c:/Program Files/Couchbase/Server/var/lib/couchbase/crash\\d754b25b-b14b-4f36-9d18-fa19f722e211.dmp before terminating.\n2021-11-12T01:39:27.297632-08:00 CRITICAL Stack backtrace of crashed thread:\n2021-11-12T01:39:27.299891-08:00 CRITICAL #0 c:\\Program Files\\Couchbase\\Server\\bin\\memcached.exe(FileOpsInterface::set_mprotect_enabled+10787324) [0x00007FF61B17D4C0]\n2021-11-12T01:39:27.299923-08:00 CRITICAL #1 c:\\Program Files\\Couchbase\\Server\\bin\\memcached.exe(FileOpsInterface::set_mprotect_enabled+11330637) [0x00007FF61B201F11]\n2021-11-12T01:39:27.299947-08:00 CRITICAL #2 C:\\Windows\\System32\\KERNEL32.DLL(BaseThreadInitThunk+20) [0x00007FFF53EB84D4]\n2021-11-12T01:39:27.299969-08:00 CRITICAL #3 C:\\Windows\\SYSTEM32\\ntdll.dll(RtlUserThreadStart+33) [0x00007FFF5483E8B1]"} 2021-11-12 01:39:35,657 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.101', u'tstamp': 1636709953338L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:39:13.338Z', u'text': u'Bucket "L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000" rebalance does not seem to be swap rebalance'} 2021-11-12 01:39:35,657 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1636709932817L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:38:52.817Z', u'text': u'Bucket "L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000" loaded on node \'ns_1@172.23.136.105\' in 0 seconds.'} 2021-11-12 01:39:35,657 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.101', u'tstamp': 1636709932526L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:38:52.526Z', u'text': u'Started rebalancing bucket L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000'} 2021-11-12 01:39:35,658 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1636709932271L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:38:52.271Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2021-11-12 01:39:35,658 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.101', u'tstamp': 1636709932176L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:38:52.176Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.103','ns_1@172.23.136.104',\n 'ns_1@172.23.136.101','ns_1@172.23.136.105'], EjectNodes = ['ns_1@172.23.136.102',\n 'ns_1@172.23.136.106'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = b413500a9e7ef72f63f40ab4f1e36723"} 2021-11-12 01:39:35,658 | test | ERROR | MainProcess | pool-6-thread-22 | [rest_client:print_UI_logs:2784] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1636709932038L, u'shortText': u'message', u'serverTime': u'2021-11-12T01:38:52.038Z', u'text': u'Node ns_1@172.23.136.105 joined cluster'} 2021-11-12 01:39:35,660 | test | ERROR | MainProcess | pool-6-thread-22 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fb98160c779f72d7dffbb5d62e873297', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=92c4ca28b447cbd9fb0b589f626bc5d7', u'status': u'notRunning'} - rebalance failed 2021-11-12 01:39:37,221 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ | L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000 | couchbase | magma | 2 | none | 0 | 2974034 | 11.72 GiB | 6.58 GiB | 3.79 GiB | 100 | +-----------------------------------------------------------------+-----------+-----------------+----------+------------+-----+---------+-----------+----------+-----------+-----+ 2021-11-12 01:39:41,703 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000, Active Resident ratio(DGM): 100% 2021-11-12 01:39:41,703 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000, Replica Resident ratio(DGM): 100% 2021-11-12 01:40:09,892 | test | ERROR | MainProcess | pool-6-thread-8 | [task:execute_tasks:3922] Failed to load 40 docs from 0 to 3000 of thread_name LoadDocs_MutateDocsFromSpecTask_1636709934.02_L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%jkb-26-12000_t7R2SMsnumf9to5%nbkd0cGjCVTPz2Er3_z%Ps0In2Qal7giNUmfxn0xoGz-26-33000_4hf6S2W6zwFEHRqdzJtgWrXoZ%K0na-IHlk5%ZsCHVZsvNDYWgGY8SCTfWBJMS9Qj-lvBChMLz3Czj9J%EZajd6SWqzI%eimy1XGs0kVk0ZEN%8xiTqnFlz-2oTGCCG-upA%2o0pd7YZQGr-26-35000_ttl=0_delete__0_3000_1636709934.28 2021-11-12 01:40:20,533 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1134] Running cbcollect on node ns_1@172.23.136.101,ns_1@172.23.136.102,ns_1@172.23.136.103,ns_1@172.23.136.104,ns_1@172.23.136.105,ns_1@172.23.136.106 2021-11-12 01:40:30,599 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1137] ns_1@172.23.136.101,ns_1@172.23.136.102,ns_1@172.23.136.103,ns_1@172.23.136.104,ns_1@172.23.136.105,ns_1@172.23.136.106 - cbcollect status: True 2021-11-12 01:40:30,599 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1142] Polling active_tasks to check cbcollect status 2021-11-12 01:45:41,364 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.136.101: Copying cbcollect ZIP file to Client 2021-11-12 01:45:44,003 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.136.102: Copying cbcollect ZIP file to Client 2021-11-12 01:45:46,325 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.136.103: Copying cbcollect ZIP file to Client 2021-11-12 01:45:48,638 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.136.104: Copying cbcollect ZIP file to Client 2021-11-12 01:45:50,937 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.136.105: Copying cbcollect ZIP file to Client 2021-11-12 01:45:52,535 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.136.106: Copying cbcollect ZIP file to Client 2021-11-12 01:45:54,065 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:594] Skipping get_trace !! 2021-11-12 01:46:24,323 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.136.101:8091/pools/default/buckets/L5VT2_xKXtFBcLfd_QevK4bT0zv3QiDW6IaZPG42CevEkRkGoe%25jkb-26-12000 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-11-12 01:46:25,252 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3101] wc: '/cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/data': Is a directory 2021-11-12 01:46:25,253 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-11-12 01:47:28,964 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1568] Rebalance done. Taken 63.4449999332 seconds to complete 2021-11-12 01:47:28,974 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete