2021-10-10 01:11:08,273 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:11:08,776 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:11:09,042 | test | INFO | MainProcess | MainThread | [basetestcase:initialize_cluster:390] Initializing cluster : C1 2021-10-10 01:11:13,144 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.130:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:15,157 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.130:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:17,167 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.130:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:19,180 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.130:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:21,193 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.130:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:26,464 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.232:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:28,477 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.232:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:30,490 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.232:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:32,502 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.232:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:34,515 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.232:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:41,234 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.252:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:43,246 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.252:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:45,259 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.252:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:47,269 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.252:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:54,756 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.76:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:56,770 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.76:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:11:58,782 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.76:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:00,795 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.76:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:07,112 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.216:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:09,125 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.216:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:11,137 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.216:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:13,147 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.216:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:15,158 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.216:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:20,950 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.129:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:22,963 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.129:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:24,974 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.129:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:26,986 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.129:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:29,000 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.106.129:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:37,348 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.15:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:39,361 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.15:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:41,372 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.15:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:43,384 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:826] Socket error while connecting to http://172.23.104.15:8091/nodes/self error [Errno 111] Connection refused 2021-10-10 01:12:59,598 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:673] RAM quota was defined less than 100 MB: 2021-10-10 01:13:13,032 | test | INFO | MainProcess | MainThread | [basetestcase:initialize_cluster:415] Cluster C1 initialized 2021-10-10 01:13:13,032 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:13:13,055 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= ClusterSetup setup started for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:13:26,608 | test | INFO | MainProcess | pool-5-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.130 | kv | 7.1.0-1458-enterprise | 2.21216691805 | Cluster node | | 172.23.104.232 | None | | | <--- IN --- | | 172.23.104.252 | None | | | <--- IN --- | | 172.23.104.76 | None | | | <--- IN --- | | 172.23.104.216 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-10-10 01:13:26,621 | test | INFO | MainProcess | pool-5-thread-8 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-10-10 01:13:41,664 | test | INFO | MainProcess | pool-5-thread-8 | [task:check:384] Rebalance - status: none, progress: 100 2021-10-10 01:13:41,680 | test | INFO | MainProcess | pool-5-thread-8 | [task:check:443] Rebalance completed with progress: 100% in 15.0720000267 sec 2021-10-10 01:13:41,813 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ | 172.23.104.232 | kv | 1.11766922014 | 11.45 GiB | 10.80 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-1458-enterprise | | 172.23.104.216 | kv | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.1.0-1458-enterprise | | 172.23.104.252 | kv | 0.200501253133 | 11.45 GiB | 10.77 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-1458-enterprise | | 172.23.106.130 | kv | 0.805740903941 | 11.45 GiB | 10.72 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-1458-enterprise | | 172.23.104.76 | kv | 0.213139418255 | 11.45 GiB | 10.59 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-1458-enterprise | +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ 2021-10-10 01:13:41,815 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= ClusterSetup setup complete for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:13:41,815 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= CollectionBase setup started for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:13:42,009 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1883] Creating required buckets from template 2021-10-10 01:13:42,520 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2473] Waiting for all collections to be created 2021-10-10 01:13:42,924 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 4.26 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ 2021-10-10 01:13:42,926 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2021-10-10 01:13:55,444 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4849] Performing scope/collection specific operations 2021-10-10 01:13:55,453 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4939] Done Performing scope/collection specific operations 2021-10-10 01:38:24,490 | test | INFO | MainProcess | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................*..................*****X | 650.000 | | 132.600 | ................X | 132.600 | | 0.000 | *************************************************************...............................................................................*.....................******X | 750.000 | | 232.600 | ..............................X | 232.600 | | 110.200 | .............******X | 110.200 | | 70.400 | ........X | 70.400 | | 39.800 | ....*****X | 39.800 | | 19.500 | .X | 19.500 | | 0.000 | ****************************************************************************..........................*........******X | 260.200 | | 220.400 | ............................X | 220.400 | | 189.800 | ........................******X | 189.800 | | 30.600 | ...X | 30.600 | | 0.000 | ********************************************************************************************...............*.....*******.*******X | 160.200 | | 139.800 | .................******X | 139.800 | | 0.000 | ****************************************************************************.....*.*******................*......******X | 210.200 | | 90.800 | ...........X | 90.800 | | 50.000 | .....******.....*******X | 89.800 | | 10.200 | X | 10.200 | | 0.000 | ****************************************************************************************************.....*....******................X | 189.800 | | 179.600 | ......................********...******X | 200.000 | | 40.800 | ....X | 40.800 | | 0.000 | *******************************************************************************************************************..........*...***************..........*.....*****X | 210.200 | | 130.600 | ................X | 130.600 | | 100.000 | ............*******X | 100.000 | | 89.800 | ..........******X | 89.800 | | 50.000 | .....*******X | 50.000 | | 10.200 | X | 10.200 | | 0.000 | *******************************************************************************************.....*.*******...........*...*****X | 150.000 | | 70.400 | ........X | 70.400 | | 50.000 | .....******X | 50.000 | | 10.200 | X | 10.200 | | 0.000 | ***************************************************************************************************..........................*........******X | 260.200 | | 220.400 | ............................X | 220.400 | | 189.800 | ........................*****X | 189.800 | | 150.600 | ...................X | 150.600 | | 0.000 | ***************..........*...*****X | 100.000 | | 20.400 | .X | 20.400 | | 0.000 | ***********************************************************************************..........*...******X | 100.000 | | 20.400 | .X | 20.400 | | 0.000 | ******...............*.....******X | 150.000 | | 30.600 | ...X | 30.600 | | 0.000 | *********************************************************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-10-10 01:38:24,546 | test | INFO | MainProcess | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.........................*......*****X | 300.000 | | 61.200 | .....X | 61.200 | | 0.000 | *************************************************************...............................................................................*.....................*****X | 950.000 | | 193.800 | ...................X | 193.800 | | 30.600 | ..*******............X | 150.000 | | 119.400 | ...........******X | 119.400 | | 0.000 | *****************************************************************************.............................*.......*****X | 350.000 | | 230.600 | .......................X | 230.600 | | 200.000 | ....................******X | 200.000 | | 40.800 | ...X | 40.800 | | 0.000 | ********************************************************************************************........*..*******.........................*.........*****X | 420.400 | | 221.400 | ......................X | 221.400 | | 129.600 | ............******X | 129.600 | | 10.200 | X | 10.200 | | 0.000 | ****************************************************************************....*.******........................*.......******X | 350.800 | | 170.600 | ................X | 170.600 | | 99.200 | .........******X | 99.200 | | 89.800 | ........*.******X | 100.000 | | 20.400 | .X | 20.400 | | 0.000 | ***************************************************************************************************....*...*******....X | 110.200 | | 79.600 | .......*******....*...******X | 150.000 | | 30.600 | ..X | 30.600 | | 0.000 | ***********************************************************************************************************....*...*******............X | 187.900 | | 181.500 | ..................*****X | 181.500 | | 62.100 | .....*X | 62.100 | | 50.000 | ....******..........*.....******X | 196.100 | | 165.100 | ................X | 165.100 | | 124.300 | ............******X | 124.300 | | 100.000 | .........X | 100.000 | | 79.600 | .......*****X | 79.600 | | 0.000 | ****************************************************************************************************....*.******X | 50.000 | | 10.200 | X | 10.200 | | 0.000 | **************..........*...******X | 125.600 | | 55.000 | ....X | 55.000 | | 24.400 | .*******..*.******X | 50.000 | | 10.200 | X | 10.200 | | 0.000 | ***************************************************************************........*..**************X | 100.000 | | 50.000 | ....*******........*...******X | 160.200 | | 80.600 | .......X | 80.600 | | 39.800 | ...*****X | 39.800 | | 0.000 | ***************************************************************************************************........*..******X | 100.000 | | 20.400 | .X | 20.400 | | 0.000 | *********************************************************************X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-10-10 01:38:24,819 | test | INFO | MainProcess | pool-5-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****.................................*.............******...............................................*.......******X | 62534.100 | | 33582.800 | ....................................................X | 33582.800 | | 27322.300 | ..........................................******X | 27322.300 | | 26444.000 | .........................................X | 26444.000 | | 25971.500 | ........................................******X | 25971.500 | | 23280.500 | ....................................X | 23280.500 | | 23262.000 | ....................................******............*...*******.........*...*****X | 40411.500 | | 36704.100 | .........................................................X | 36704.100 | | 34874.700 | ......................................................*******.*.******.......X | 40265.200 | | 39625.700 | ..............................................................******X | 39625.700 | | 16747.200 | .........................X | 16747.200 | | 11445.300 | .................******...................*.......******X | 27728.100 | | 25477.400 | .......................................X | 25477.400 | | 23741.500 | ....................................******X | 23741.500 | | 20778.000 | ................................*.******.............*..******X | 30956.200 | | 28171.900 | ............................................X | 28171.900 | | 26554.500 | .........................................*****X | 26554.500 | | 24603.700 | ......................................********..........*..*****X | 32416.000 | | 25605.000 | .......................................X | 25605.000 | | 24270.200 | .....................................******X | 24270.200 | | 21134.300 | ................................X | 21134.300 | | 20195.200 | ...............................******.........................*......******X | 39872.200 | | 32377.800 | ..................................................X | 32377.800 | | 29816.700 | ..............................................******X | 29816.700 | | 21108.000 | ................................X | 21108.000 | | 21037.900 | ................................******............................*...******X | 40505.400 | | 21648.400 | .................................X | 21648.400 | | 17049.500 | ..........................******X | 17049.500 | | 12003.600 | ..................*.******........................*..******X | 29372.400 | | 20919.100 | ................................X | 20919.100 | | 19350.800 | .............................*******..*.******.........*.*******....*..*****X | 30746.100 | | 20275.800 | ...............................X | 20275.800 | | 16669.100 | .........................*******..............*.....******X | 28315.600 | | 23864.600 | .....................................X | 23864.600 | | 22550.600 | ...................................*****X | 22550.600 | | 21551.200 | .................................X | 21551.200 | | 21365.200 | .................................******X | 21365.200 | | 21301.900 | .................................X | 21301.900 | | 20499.700 | ...............................*******...........*....*****X | 29641.600 | | 19711.100 | ..............................X | 19711.100 | | 17162.300 | ..........................*******.................*...*****X | 29968.400 | | 28288.000 | ............................................*..******X | 29559.500 | | 22452.600 | ..................................X | 22452.600 | | 20582.600 | ...............................******...................*..******X | 33464.900 | | 22264.700 | ..................................X | 22264.700 | | 20469.800 | ...............................*******......................******X | 34393.300 | | 32824.000 | ...................................................X | 32824.000 | | 30732.900 | ................................................*****X | 30732.900 | | 26387.000 | .........................................X | 26387.000 | | 26079.200 | ........................................******X | 26079.200 | | 19730.100 | ..............................X | 19730.100 | | 19124.900 | .............................******..............*.******X | 28545.000 | | 22099.200 | ..................................X | 22099.200 | | 20764.200 | ................................********.******.........*******X | 27512.500 | | 21037.000 | ................................X | 21037.000 | | 19401.500 | ..............................*****X | 19401.500 | | 19133.500 | .............................********................*....******X | 31725.100 | | 24999.300 | ......................................X | 24999.300 | | 23415.100 | ....................................*****X | 23415.100 | | 16961.700 | ..........................X | 16961.700 | | 15186.800 | .......................*******..................*....*****X | 28931.500 | | 20618.800 | ...............................X | 20618.800 | | 18636.300 | ............................*******.......*...******.X | 25489.500 | | 24195.600 | .....................................******X | 24195.600 | | 18398.000 | ............................X | 18398.000 | | 17331.400 | ..........................*******.......******X | 21544.800 | | 20857.700 | ................................X | 20857.700 | | 20073.500 | ...............................*******...*.******X | 22779.000 | | 13572.100 | ....................X | 13572.100 | | 11592.300 | .................******...................*....******X | 25867.500 | | 18794.100 | .............................X | 18794.100 | | 16928.400 | ..........................******X | 16928.400 | | 11042.200 | ................X | 11042.200 | | 10562.800 | ...............******.............................*.....******X | 31417.900 | | 15701.400 | ........................X | 15701.400 | | 12295.800 | ..................******..*..*******...............*..*****X | 25484.500 | | 23670.600 | ....................................X | 23670.600 | | 23307.400 | ....................................******X | 23307.400 | | 10054.500 | ...............X | 10054.500 | | 7127.400 | ..........*******..........................*......******X | 27265.000 | | 20463.500 | ...............................X | 20463.500 | | 18345.800 | ............................*****X | 18345.800 | | 17163.900 | ..........................*.*******.X | 18226.000 | | 17043.100 | ..........................******...*...******X | 21111.000 | | 14952.400 | ......................X | 14952.400 | | 12916.500 | ...................*******.....................*.....*****X | 29303.800 | | 20843.900 | ................................X | 20843.900 | | 18006.300 | ...........................******X | 18006.300 | | 9824.900 | ..............X | 9824.900 | | 9122.700 | .............*******.....................*.....*****X | 25175.400 | | 21317.200 | .................................X | 21317.200 | | 19755.500 | ..............................******X | 19755.500 | | 10651.600 | ................X | 10651.600 | | 8201.500 | ............*******...............*.....*****X | 20984.800 | | 15887.200 | ........................X | 15887.200 | | 13848.400 | .....................*******.*.*******.................*....*****X | 28243.400 | | 23968.500 | .....................................X | 23968.500 | | 23185.200 | ....................................******X | 23185.200 | | 17565.600 | ...........................X | 17565.600 | | 15858.300 | ........................******..................*.....******X | 30128.400 | | 23552.500 | ....................................X | 23552.500 | | 21722.600 | .................................*****X | 21722.600 | | 14025.000 | .....................X | 14025.000 | | 11910.700 | ..................*******..........*....******..X | 22314.600 | | 22095.100 | ..................................******X | 22095.100 | | 15711.900 | ........................X | 15711.900 | | 14994.900 | ......................*******.............*.*****X | 23474.300 | | 19172.400 | .............................*******X | 19230.700 | | 18002.700 | ...........................X | 18002.700 | | 17548.400 | ...........................*******...........*..*****X | 26198.600 | | 20922.300 | ................................X | 20922.300 | | 19185.600 | .............................******X | 19185.600 | | 11967.000 | ..................X | 11967.000 | | 11113.900 | ................******.................*....******X | 24094.200 | | 22866.500 | ...................................X | 22866.500 | | 21858.500 | .................................******X | 21858.500 | | 14228.100 | .....................X | 14228.100 | | 12542.600 | ...................******.....*..*******....*******X | 19761.500 | | 14821.300 | ......................X | 14821.300 | | 13769.900 | .....................*******.......................*.......*****X | 33128.100 | | 29041.800 | .............................................X | 29041.800 | | 26743.500 | .........................................******X | 26743.500 | | 20171.100 | ...............................X | 20171.100 | | 18893.400 | .............................*******..............*.....*****X | 31014.300 | | 25109.000 | .......................................X | 25109.000 | | 22852.600 | ...................................********.******.........*...******X | 30981.700 | | 29718.500 | ..............................................X | 29718.500 | | 28090.600 | ...........................................******X | 28090.600 | | 20700.900 | ................................X | 20700.900 | | 20099.000 | ...............................******............*..******X | 28981.800 | | 21496.600 | .................................X | 21496.600 | | 19760.200 | ..............................******X | 19760.200 | | 13722.800 | ....................X | 13722.800 | | 12607.200 | ...................******........................*......******X | 31514.100 | | 25397.600 | .......................................X | 25397.600 | | 23875.100 | .....................................******X | 23875.100 | | 13934.000 | .....................X | 13934.000 | | 12065.900 | ..................******....................*....******X | 27348.400 | | 23434.400 | ....................................X | 23434.400 | | 22430.600 | ..................................******X | 22430.600 | | 14197.200 | .....................X | 14197.200 | | 12565.300 | ...................******............*...*************X | 22158.200 | | 19341.800 | .............................X | 19341.800 | | 19031.900 | .............................*******.............*...*****X | 28923.200 | | 22116.900 | ..................................X | 22116.900 | | 19496.200 | ..............................******X | 19496.200 | | 15534.500 | .......................********...........*..******....X | 25849.800 | | 24768.200 | ......................................******X | 24768.200 | | 17314.600 | ..........................X | 17314.600 | | 16892.800 | ..........................*******........*..******.........*******X | 29387.300 | | 19245.600 | .............................X | 19245.600 | | 17410.400 | ..........................*******....X | 19592.200 | | 19533.700 | ..............................******.....*..******X | 24325.100 | | 17454.300 | ..........................X | 17454.300 | | 15764.600 | ........................******..........*..******X | 23654.600 | | 18311.100 | ............................X | 18311.100 | | 17909.700 | ...........................******X | 17909.700 | | 13929.800 | .....................X | 13929.800 | | 12690.700 | ...................*******.............*...X | 22790.700 | +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-10-10 01:38:26,190 | test | INFO | MainProcess | pool-5-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 22790.700 | ...................................................................................................X | 22790.700 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2021-10-10 01:38:26,351 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+---------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+---------------------+--------------------+-----------------------+ | 172.23.104.232 | kv | 46.0808210391 | 11.45 GiB | 7.27 GiB | 0.0 Byte / 3.50 GiB | 6526221 / 13019902 | 7.1.0-1458-enterprise | | 172.23.104.216 | kv | 46.5823431711 | 11.45 GiB | 7.19 GiB | 0.0 Byte / 3.50 GiB | 6506627 / 12997769 | 7.1.0-1458-enterprise | | 172.23.104.252 | kv | 46.5861411316 | 11.45 GiB | 8.34 GiB | 0.0 Byte / 3.50 GiB | 6425390 / 12883805 | 7.1.0-1458-enterprise | | 172.23.106.130 | kv | 46.1459271651 | 11.45 GiB | 7.21 GiB | 0.0 Byte / 3.50 GiB | 6527555 / 13078923 | 7.1.0-1458-enterprise | | 172.23.104.76 | kv | 46.9756780848 | 11.45 GiB | 8.44 GiB | 0.0 Byte / 3.50 GiB | 6419053 / 12818334 | 7.1.0-1458-enterprise | +----------------+----------+-----------------+-----------+----------+---------------------+--------------------+-----------------------+ 2021-10-10 01:38:56,654 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4603] Validating collection stats and item counts 2021-10-10 01:39:14,992 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:validate_manifest_uid:4466] Bucket UID mismatch. Expected: 0, Actual: 1 2021-10-10 01:39:14,993 | test | WARNING | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4620] Bucket manifest UID mismatch! 2021-10-10 01:39:29,299 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:validate_manifest_uid:4466] Bucket UID mismatch. Expected: 0, Actual: 1 2021-10-10 01:39:29,301 | test | WARNING | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4620] Bucket manifest UID mismatch! 2021-10-10 01:39:43,500 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:validate_manifest_uid:4466] Bucket UID mismatch. Expected: 0, Actual: 1 2021-10-10 01:39:43,502 | test | WARNING | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4620] Bucket manifest UID mismatch! 2021-10-10 01:40:06,456 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 250.87 MiB | 319.09 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 525.89 MiB | 557.91 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.77 GiB | 39.71 GiB | 3.42467843438 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2021-10-10 01:40:06,457 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:621] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_swap_rebalance ========= 2021-10-10 01:40:12,459 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:894] Doing collection data load during swap_rebalance 2021-10-10 01:40:12,460 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:355] Starting rebalance operation of type : swap_rebalance 2021-10-10 01:40:19,696 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait for rebalance to start 2021-10-10 01:40:19,831 | test | INFO | MainProcess | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.104.15 | kv | 7.1.0-1458-enterprise | 0 | Cluster node | | 172.23.104.232 | kv | 7.1.0-1458-enterprise | 8.14217292664 | Cluster node | | 172.23.106.129 | kv | 7.1.0-1458-enterprise | 0 | Cluster node | | 172.23.104.216 | kv | 7.1.0-1458-enterprise | 9.16887375457 | --- OUT ---> | | 172.23.104.252 | kv | 7.1.0-1458-enterprise | 8.40590685346 | Cluster node | | 172.23.106.130 | kv | 7.1.0-1458-enterprise | 10.6821921276 | Cluster node | | 172.23.104.76 | kv | 7.1.0-1458-enterprise | 8.25444907232 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-10-10 01:40:19,842 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 0.0 2021-10-10 01:40:29,696 | test | INFO | MainProcess | MainThread | [collections_rebalance:compact_all_buckets:238] Starting compaction for each bucket 2021-10-10 01:40:29,845 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4849] Performing scope/collection specific operations 2021-10-10 01:40:29,855 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4939] Done Performing scope/collection specific operations 2021-10-10 01:40:29,881 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 17.25 2021-10-10 01:40:39,921 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 32.03 2021-10-10 01:40:49,964 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 33.33 2021-10-10 01:41:00,012 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 37.69 2021-10-10 01:41:10,053 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 47.21 2021-10-10 01:41:20,092 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 55.36 2021-10-10 01:41:30,132 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 62.35 2021-10-10 01:41:40,171 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 64.53 2021-10-10 01:41:50,216 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 66.67 2021-10-10 01:42:00,252 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 66.67 2021-10-10 01:42:10,293 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 67.33 2021-10-10 01:42:20,335 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 67.77 2021-10-10 01:42:30,374 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 68.39 2021-10-10 01:42:40,414 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 68.91 2021-10-10 01:42:47,529 | test | INFO | MainProcess | pool-5-thread-13 | [task:call:5735] bucket1 compaction done: 100% 2021-10-10 01:42:50,454 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 69.26 2021-10-10 01:43:00,493 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 69.93 2021-10-10 01:43:10,536 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 70.43 2021-10-10 01:43:20,575 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 71.05 2021-10-10 01:43:30,614 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 71.63 2021-10-10 01:43:40,651 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 72.42 2021-10-10 01:43:50,690 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 73.09 2021-10-10 01:44:00,727 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 73.67 2021-10-10 01:44:10,765 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 74.4 2021-10-10 01:44:20,802 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 75.52 2021-10-10 01:44:30,844 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 76.39 2021-10-10 01:44:40,880 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 77.07 2021-10-10 01:44:50,920 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 77.69 2021-10-10 01:45:00,957 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 78.37 2021-10-10 01:45:11,000 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 78.56 2021-10-10 01:45:21,040 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 79.3 2021-10-10 01:45:31,082 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 79.66 2021-10-10 01:45:41,122 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 79.76 2021-10-10 01:45:51,161 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 79.94 2021-10-10 01:46:01,201 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 80.31 2021-10-10 01:46:11,242 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 80.53 2021-10-10 01:46:21,285 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 80.81 2021-10-10 01:46:31,328 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.05 2021-10-10 01:46:41,371 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.23 2021-10-10 01:46:51,414 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.23 2021-10-10 01:47:01,454 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.31 2021-10-10 01:47:11,497 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.34 2021-10-10 01:47:21,536 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.38 2021-10-10 01:47:31,578 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.41 2021-10-10 01:47:41,637 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.41 2021-10-10 01:47:51,677 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.44 2021-10-10 01:48:01,716 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.49 2021-10-10 01:48:11,759 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.49 2021-10-10 01:48:21,812 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.61 2021-10-10 01:48:31,852 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.61 2021-10-10 01:48:41,894 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.68 2021-10-10 01:48:51,937 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.71 2021-10-10 01:49:01,977 | test | INFO | MainProcess | pool-5-thread-6 | [task:check:384] Rebalance - status: running, progress: 81.71 2021-10-10 01:49:12,023 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:_rebalance_status_and_progress:1638] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'05d3eb5f94772cb5477d3fcb95d71cff', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=94335be55ba0e3a6d20f1438e03535b4', u'status': u'notRunning'} - rebalance failed 2021-10-10 01:49:12,055 | test | INFO | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2784] Latest logs from UI on 172.23.106.130: 2021-10-10 01:49:12,056 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.106.130', u'tstamp': 1633855747820L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:49:07.820Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.27537.16>,\n {{wait_seqno_persisted_failed,"default",165,\n 33468,\n [{\'ns_1@172.23.104.232\',\n {\'EXIT\',\n {{badarg,\n [{erlang,link,[undefined],[]},\n {janitor_agent,\n \'-do_handle_call/3-fun-5-\',3,\n [{file,"src/janitor_agent.erl"},\n {line,743}]},\n {janitor_agent,\n \'-spawn_rebalance_subprocess/3-fun-0-\',\n 3,\n [{file,"src/janitor_agent.erl"},\n {line,938}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.104.232\'},\n {if_rebalance,<0.21509.13>,\n {wait_seqno_persisted,165,33468}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,506}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 28c564fc85e03e205d3b703a88dfb285'} 2021-10-10 01:49:12,056 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.106.130', u'tstamp': 1633855747798L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:49:07.798Z', u'text': u'Worker <0.22102.16> (for action {move,{165,\n [\'ns_1@172.23.104.216\',\n \'ns_1@172.23.106.130\',\n \'ns_1@172.23.104.232\'],\n [\'ns_1@172.23.104.15\',\n \'ns_1@172.23.106.130\',\n \'ns_1@172.23.104.232\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.27537.16>,\n {{wait_seqno_persisted_failed,\n "default",\n 165,33468,\n [{\'ns_1@172.23.104.232\',\n {\'EXIT\',\n {{badarg,\n [{erlang,\n link,\n [undefined],\n []},\n {janitor_agent,\n \'-do_handle_call/3-fun-5-\',\n 3,\n [{file,\n "src/janitor_agent.erl"},\n {line,\n 743}]},\n {janitor_agent,\n \'-spawn_rebalance_subprocess/3-fun-0-\',\n 3,\n [{file,\n "src/janitor_agent.erl"},\n {line,\n 938}]},\n {proc_lib,\n init_p,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.104.232\'},\n {if_rebalance,\n <0.21509.13>,\n {wait_seqno_persisted,\n 165,\n 33468}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 506}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-10-10 01:49:12,058 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.232', u'tstamp': 1633855746510L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:49:06.510Z', u'text': u'Bucket "bucket1" loaded on node \'ns_1@172.23.104.232\' in 0 seconds.'} 2021-10-10 01:49:12,058 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.104.232', u'tstamp': 1633855745400L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:49:05.400Z', u'text': u"Service 'memcached' exited with status 139. Restarting. Messages:\n #16 /opt/couchbase/bin/memcached(_ZN5magma5Magma14CompactKVStoreEtRKNS_5SliceES3_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS6_EEtEE+0x6d) [0x400000+0x51239d]\n #17 /opt/couchbase/bin/memcached() [0x400000+0x47766e]\n #18 /opt/couchbase/bin/memcached() [0x400000+0x46e780]\n #19 /opt/couchbase/bin/memcached() [0x400000+0x46ed06]\n #20 /opt/couchbase/bin/memcached() [0x400000+0x406eca]\n #21 /opt/couchbase/bin/memcached() [0x400000+0x4089b1]\n #22 /opt/couchbase/bin/memcached() [0x400000+0x314096]\n #23 /opt/couchbase/bin/memcached() [0x400000+0x654332]\n #24 /opt/couchbase/bin/memcached() [0x400000+0x6514d5]\n #25 /opt/couchbase/bin/memcached() [0x400000+0x7a51e0]\n #26 /opt/couchbase/bin/memcached() [0x400000+0x78cf9a]\n #27 /opt/couchbase/bin/memcached() [0x400000+0x7a8199]\n #28 /opt/couchbase/bin/memcached() [0x400000+0x651164]\n #29 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb3541da000+0xcdd40]\n #30 /lib64/libpthread.so.0() [0x7fb3560a8000+0x7ea5]\n #31 /lib64/libc.so.6(clone+0x6d) [0x7fb3538f2000+0xfe8dd]"} 2021-10-10 01:49:12,058 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.104.232', u'tstamp': 1633855745376L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:49:05.376Z', u'text': u"Control connection to memcached on 'ns_1@172.23.104.232' disconnected. Check logs for details."} 2021-10-10 01:49:12,059 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'compaction_daemon', u'type': u'info', u'node': u'ns_1@172.23.104.252', u'tstamp': 1633855637816L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:47:17.816Z', u'text': u'User-triggered compaction of bucket `bucket2` completed.'} 2021-10-10 01:49:12,059 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'compaction_daemon', u'type': u'info', u'node': u'ns_1@172.23.106.130', u'tstamp': 1633855591043L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:46:31.043Z', u'text': u'User-triggered compaction of bucket `default` completed.'} 2021-10-10 01:49:12,059 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'compaction_daemon', u'type': u'info', u'node': u'ns_1@172.23.104.252', u'tstamp': 1633855365265L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:42:45.265Z', u'text': u'User-triggered compaction of bucket `bucket1` completed.'} 2021-10-10 01:49:12,059 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.106.130', u'tstamp': 1633855321242L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:42:01.242Z', u'text': u'Bucket "default" rebalance appears to be swap rebalance'} 2021-10-10 01:49:12,059 | test | ERROR | MainProcess | pool-5-thread-6 | [rest_client:print_UI_logs:2786] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.129', u'tstamp': 1633855319855L, u'shortText': u'message', u'serverTime': u'2021-10-10T01:41:59.855Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.129\' in 0 seconds.'} 2021-10-10 01:49:12,059 | test | ERROR | MainProcess | pool-5-thread-6 | [task:call:296] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'05d3eb5f94772cb5477d3fcb95d71cff', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=94335be55ba0e3a6d20f1438e03535b4', u'status': u'notRunning'} - rebalance failed 2021-10-10 01:49:13,125 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 49580 | 9.77 GiB | 255.55 MiB | 266.23 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 49680 | 4.88 GiB | 453.21 MiB | 367.15 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32485894 | 3.50 GiB | 2.68 GiB | 36.21 GiB | 7.41296471674 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2021-10-10 01:49:13,865 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-10-10 01:49:13,867 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-10-10 01:49:30,801 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-10-10 01:49:30,801 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-10-10 01:49:37,586 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 5% 2021-10-10 01:49:37,586 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 0% 2021-10-10 01:49:59,421 | test | ERROR | MainProcess | pool-5-thread-15 | [task:execute_tasks:3922] Failed to load 70 docs from 26060 to 46908 of thread_name LoadDocs_MutateDocsFromSpecTask_1633855234.12_default_gR%n%dKIXdUwFccfdz6J62TXu%1dmID91Ok8UzDbd9NTvw_GV5ZFFX2F-GF5Pi-XzFv4bW9aJUDCJmvUT-5SsU4gJXJYeDmf8no24e13Gz1NiHRZshXSr0pZWOwB5_zl2FLcDVfYhmGvOLU27yocn982pBt7Pg7BQuT7L2IaFSikL0cXnzcBuzbbLvGDZB0of-pzu-LoUse4qk9rgAU3--42-66000_RYZcg9zYObD7ci_KQkCAMtUuJuCR%nvagxLSuZnOcI9BshsG4_qfGafRoLfk1sGTQkZ0ttNfgPuadfXxmYgQXGDMo-pjs85OC7oGLMsAiZ-6bHjGFcEg9QpakzwwOV2juP6MsYQ5iMBb-NuqJnmEeuHUvGig0acg2e3sCatGwtmZQyo3vufQdHGixM7kMhGJSJWiKYH6iSCg0geru9-xB3sC-42-67000_ttl=0_read__26060_46908_1633855234.4 2021-10-10 01:50:04,421 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:71] Pool did not terminate 2021-10-10 01:50:06,423 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:71] Pool did not terminate 2021-10-10 01:50:11,394 | test | INFO | MainProcess | MainThread | [basetestcase:check_logs:795] Comparing timestamps: Log's latest timestamp: 2021-10-10 01:49:05, Test's start timestamp is 2021-10-10 01:11:08.272000 2021-10-10 01:50:11,395 | test | INFO | MainProcess | MainThread | [basetestcase:check_logs:795] Comparing timestamps: Log's latest timestamp: 2021-10-10 01:49:05, Test's start timestamp is 2021-10-10 01:11:08.272000 2021-10-10 01:50:11,395 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:904] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-10-10 01:50:11,397 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:906] 172.23.104.232: Found ' CRITICAL ' logs - ['[ns_server:info,2021-10-10T01:49:05.399-07:00,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1458). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/29e3e32c-fd4c-4398-418dc5bf-529aecdc.dmp before terminating.\n', '[ns_server:info,2021-10-10T01:49:05.660-07:00,babysitter_of_ns_1@cb.local:<0.867.0>:ns_port_server:log:221]memcached<0.867.0>: 2021-10-10T01:49:05.460178-07:00 CRITICAL Detected previous crash\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460225-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1458). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/29e3e32c-fd4c-4398-418dc5bf-529aecdc.dmp before terminating.\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460236-07:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460237-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x6d5fc8]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460239-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x7260da]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460241-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x726418]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460242-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fb3560a8000+0xf630]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460244-07:00 CRITICAL #4 /opt/couchbase/bin/memcached() [0x400000+0x56f450]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460245-07:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x5663cf]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460257-07:00 CRITICAL #6 /opt/couchbase/bin/memcached() [0x400000+0x5675e4]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460259-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x567fc8]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460260-07:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x5687cd]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460264-07:00 CRITICAL #9 /opt/couchbase/bin/memcached() [0x400000+0x568cc8]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460294-07:00 CRITICAL #10 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl11syncKVStoreEtb+0x215) [0x400000+0x511885]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460296-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x511a60]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460297-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x51838e]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460306-07:00 CRITICAL #13 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtNS0_9StoreTypeESt8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS5_EEtEE+0x36e) [0x400000+0x511f0e]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460308-07:00 CRITICAL #14 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtNS0_9StoreTypeESt8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS5_EEtEE+0xff) [0x400000+0x511c9f]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460310-07:00 CRITICAL #15 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtRKNS_5SliceES4_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS7_EEtEE+0x65) [0x400000+0x512095]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460312-07:00 CRITICAL #16 /opt/couchbase/bin/memcached(_ZN5magma5Magma14CompactKVStoreEtRKNS_5SliceES3_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS6_EEtEE+0x6d) [0x400000+0x51239d]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460336-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x47766e]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460340-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x46e780]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460341-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x46ed06]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460345-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x406eca]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460370-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x4089b1]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460372-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x314096]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460374-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x654332]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460397-07:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x6514d5]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460426-07:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x7a51e0]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460427-07:00 CRITICAL #26 /opt/couchbase/bin/memcached() [0x400000+0x78cf9a]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460429-07:00 CRITICAL #27 /opt/couchbase/bin/memcached() [0x400000+0x7a8199]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460430-07:00 CRITICAL #28 /opt/couchbase/bin/memcached() [0x400000+0x651164]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460431-07:00 CRITICAL #29 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb3541da000+0xcdd40]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460432-07:00 CRITICAL #30 /lib64/libpthread.so.0() [0x7fb3560a8000+0x7ea5]\n', 'memcached<0.867.0>: 2021-10-10T01:49:05.460434-07:00 CRITICAL #31 /lib64/libc.so.6(clone+0x6d) [0x7fb3538f2000+0xfe8dd]\n'] 2021-10-10 01:50:14,970 | test | INFO | MainProcess | MainThread | [basetestcase:check_logs:795] Comparing timestamps: Log's latest timestamp: 2021-10-10 08:49:23, Test's start timestamp is 2021-10-10 01:11:08.272000 2021-10-10 01:50:14,970 | test | INFO | MainProcess | MainThread | [basetestcase:check_logs:795] Comparing timestamps: Log's latest timestamp: 2021-10-10 08:49:23, Test's start timestamp is 2021-10-10 01:11:08.272000 2021-10-10 01:50:14,970 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:904] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-10-10 01:50:14,970 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:906] 172.23.104.15: Found ' CRITICAL ' logs - ['[ns_server:info,2021-10-10T08:49:23.339Z,babysitter_of_ns_1@cb.local:<0.130.0>:ns_port_server:log:221]memcached<0.130.0>: 2021-10-10T08:49:23.138978+00:00 CRITICAL (default) MagmaKVStore::prepareToDeleteImpl vb:164 GetKVStoreRevision failed. Status:NotExists: KVStore ID:164 does not exist\n'] 2021-10-10 01:50:15,012 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1134] Running cbcollect on node ns_1@172.23.104.15,ns_1@172.23.104.216,ns_1@172.23.104.232,ns_1@172.23.104.252,ns_1@172.23.104.76,ns_1@172.23.106.129,ns_1@172.23.106.130 2021-10-10 01:50:25,043 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1137] ns_1@172.23.104.15,ns_1@172.23.104.216,ns_1@172.23.104.232,ns_1@172.23.104.252,ns_1@172.23.104.76,ns_1@172.23.106.129,ns_1@172.23.106.130 - cbcollect status: True 2021-10-10 01:50:25,045 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1142] Polling active_tasks to check cbcollect status 2021-10-10 01:55:45,908 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.104.15: Copying cbcollect ZIP file to Client 2021-10-10 01:55:47,108 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.104.216: Copying cbcollect ZIP file to Client 2021-10-10 01:55:48,601 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.104.232: Copying cbcollect ZIP file to Client 2021-10-10 01:55:50,029 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.104.252: Copying cbcollect ZIP file to Client 2021-10-10 01:55:51,542 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.104.76: Copying cbcollect ZIP file to Client 2021-10-10 01:55:52,838 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.106.129: Copying cbcollect ZIP file to Client 2021-10-10 01:55:54,198 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1169] ns_1@172.23.106.130: Copying cbcollect ZIP file to Client 2021-10-10 01:55:55,986 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:567] Skipping get_trace !! 2021-10-10 01:56:26,286 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.106.130:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-10-10 01:56:26,905 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3092] wc: /data: Is a directory 2021-10-10 01:56:26,907 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1601] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-10-10 01:56:57,111 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.106.130:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-10-10 01:56:57,681 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3092] wc: /data: Is a directory 2021-10-10 01:56:57,683 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1601] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-10-10 01:57:27,835 | infra | ERROR | MainProcess | MainThread | [Rest_Connection:_http_request:283] DELETE http://172.23.106.130:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-10-10 01:57:28,490 | infra | ERROR | MainProcess | MainThread | [remote_util:log_command_output:3092] wc: /data: Is a directory 2021-10-10 01:57:28,490 | test | ERROR | MainProcess | MainThread | [bucket_ready_functions:delete_bucket:1601] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-10-10 02:00:38,417 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1574] Rebalance done. Taken 189.610000134 seconds to complete 2021-10-10 02:00:38,431 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete