guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini rerun=False,get-cbcollect-info=False,skip_cluster_reset=True,upgrade_version=7.2.0-5284 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,doc_size=512,data_load_stage=during,bucket_history_retention_seconds=300,bucket_spec=magma_dgm.20_percent_dgm.5_node_2_replica_magma_512,bucket_history_retention_bytes=2147483648,nodes_failover=1,durability=MAJORITY,recovery_type=delta,skip_validations=False,nodes_init=5,default_history_retention_for_collections=false,disk_optimized_thread_settings=True,autoCompactionDefined=true,randomize_value=True,override_spec_params=durability,disk_optimized_thread_settings=True,get-cbcollect-info=False,autoCompactionDefined=true,dedupe_update_itrs=3000' Test Input params: {'data_load_stage': 'during', 'doc_size': '512', 'conf_file': 'debug.conf', 'upgrade_version': '7.2.0-5284', 'spec': 'debug', 'rerun': 'False', 'num_nodes': 6, 'bucket_history_retention_seconds': '300', 'bucket_spec': 'magma_dgm.20_percent_dgm.5_node_2_replica_magma_512', 'case_number': 1, 'dedupe_update_itrs': '3000', 'cluster_name': 'node', 'bucket_history_retention_bytes': '2147483648', 'nodes_failover': '1', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'durability': 'MAJORITY', 'recovery_type': 'delta', 'skip_validations': 'False', 'logs_folder': '/data/workspace/temp_job_ashwin/logs/testrunner-23-Mar-31_04-09-30/test_1', 'nodes_init': '5', 'skip_cluster_reset': 'True', 'default_history_retention_for_collections': 'false', 'disk_optimized_thread_settings': 'True', 'autoCompactionDefined': 'true', 'randomize_value': 'True', 'override_spec_params': 'durability'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2023-03-31 04:09:35,788 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:09:36,213 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:09:36,923 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:09:36,953 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= ClusterSetup setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:09:57,444 | test | INFO | pool-3-thread-13 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-----+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+-----+--------------+-----------------------+ | 172.23.107.222 | kv | 7.2.0-5284-enterprise | 0 | Cluster node | active / none | | 172.23.107.223 | None | | | <--- IN --- | | | 172.23.107.217 | None | | | <--- IN --- | | | 172.23.107.99 | None | | | <--- IN --- | | | 172.23.107.102 | None | | | <--- IN --- | | +----------------+----------+-----------------------+-----+--------------+-----------------------+ 2023-03-31 04:09:57,476 | test | INFO | pool-3-thread-13 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-03-31 04:10:12,986 | test | INFO | pool-3-thread-13 | [task:check:421] Rebalance - status: none, progress: 100 2023-03-31 04:10:13,009 | test | INFO | pool-3-thread-13 | [task:check:480] Rebalance completed with progress: 100% in 15.5639998913 sec 2023-03-31 04:10:13,180 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.107.217 | kv | 1.44801674772 | 23.36 GiB | 22.09 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.222 | kv | 2.21032824747 | 23.36 GiB | 22.11 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.102 | kv | 1.11370545048 | 23.36 GiB | 22.35 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.99 | kv | 1.00183860202 | 23.36 GiB | 22.15 GiB | 56.69 MiB / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.223 | kv | 1.28704194559 | 23.36 GiB | 22.05 GiB | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.2.0-5284-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2023-03-31 04:10:13,180 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= ClusterSetup setup complete for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:10:13,184 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= CollectionBase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:10:13,437 | test | INFO | MainThread | [collections_base:deploy_buckets_from_spec_file:317] Creating bucket from spec: magma_dgm.20_percent_dgm.5_node_2_replica_magma_512 2023-03-31 04:10:13,440 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1957] Creating required buckets from template 2023-03-31 04:10:13,862 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2689] Waiting for all collections to be created 2023-03-31 04:10:13,963 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.107.217 | kv | 1.25490985088 | 23.36 GiB | 22.09 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.222 | kv | 2.21032824747 | 23.36 GiB | 22.11 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.102 | kv | 1.11370545048 | 23.36 GiB | 22.35 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.99 | kv | 1.00183860202 | 23.36 GiB | 22.15 GiB | 56.69 MiB / 3.50 GiB | 0 / 0 | 7.2.0-5284-enterprise | | 172.23.107.223 | kv | 1.28704194559 | 23.36 GiB | 22.05 GiB | 0.0 Byte / 0.0 Byte | 0 / 0 | 7.2.0-5284-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2023-03-31 04:10:14,194 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 0.0 Byte | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ 2023-03-31 04:10:14,196 | test | INFO | MainThread | [collections_base:create_clients_for_sdk_pool:336] Creating required SDK clients for client_pool 2023-03-31 04:10:31,092 | test | INFO | MainThread | [collections_base:load_data_from_spec_file:346] Loading data using spec: initial_load 2023-03-31 04:10:31,109 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5159] Performing scope/collection specific operations 2023-03-31 04:10:31,137 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5249] Done Performing scope/collection specific operations 2023-03-31 04:15:52,565 | test | INFO | pool-3-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.*****......***...................*.....***X | 484.400 | | 384.900 | .......................***X | 384.900 | | 0.000 | ********..*...*****..............***..............................................****X | 1015.600 | | 796.700 | ..................................................**X | 796.700 | | 168.000 | .........X | 168.000 | | 0.000 | ********.......****........***................................................****X | 976.500 | | 961.900 | .............................................................**X | 961.900 | | 311.100 | ...................X | 311.100 | | 112.100 | ......***X | 112.100 | | 0.000 | ********...***.....................****X | 373.100 | | 367.200 | ......................**X | 367.200 | | 126.900 | .......****X | 126.900 | | 80.100 | ....**X | 80.100 | | 19.900 | X | 19.900 | | 0.000 | ****.****.......................................................................*****............................**X | 1542.700 | | 1175.200 | ...........................................................................****X | 1175.200 | | 737.400 | ..............................................**X | 737.400 | | 0.000 | ************......****....................................****X | 660.200 | | 620.400 | .......................................**X | 620.400 | | 139.800 | ........****X | 139.800 | | 80.100 | ....**X | 80.100 | | 0.000 | *****.....***...............................*****....**X | 620.600 | | 560.400 | ...................................X | 560.400 | | 540.500 | ..................................***X | 540.500 | | 400.500 | ........................**X | 400.500 | | 99.500 | .....X | 99.500 | | 0.000 | ********..........................****............********X | 597.700 | | 400.500 | ........................**X | 400.500 | | 0.000 | *************...***................................*****...**X | 596.200 | | 355.400 | ......................X | 355.400 | | 251.400 | ...............***X | 251.400 | | 148.000 | ........**X | 148.000 | | 27.600 | X | 27.600 | | 0.000 | ****.***.......................*........****.......***X | 604.400 | | 195.100 | ...........****X | 195.100 | | 75.700 | ...*X | 75.700 | +----------+----------------------------------------------------------------------------------------------------------------------+----------+ 2023-03-31 04:15:52,917 | test | INFO | pool-3-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------------+-----------+ | 39.000 | *****..........***.......................................................*....................*****........***.....*****..***X | 27499.200 | | 27476.600 | ..................................................................................................****X | 27476.600 | | 27083.900 | .................................................................................................**X | 27083.900 | | 25868.900 | .............................................................................................X | 25868.900 | | 25523.800 | ...........................................................................................****.****...****X | 26599.000 | | 26568.100 | ...............................................................................................*******X | 26665.000 | | 26210.600 | ..............................................................................................**X | 26210.600 | | 23257.400 | ...................................................................................*****.***.......*...********.***X | 26580.200 | | 26531.600 | ...............................................................................................***X | 26531.600 | | 25017.300 | .........................................................................................***X | 25017.300 | | 24764.000 | .........................................................................................****....*****.***..****X | 26762.300 | | 26716.400 | ................................................................................................**X | 26716.400 | | 25062.300 | ..........................................................................................****X | 25062.300 | | 24452.900 | .......................................................................................**X | 24452.900 | | 24288.100 | .......................................................................................*****.***...*.***X | 25716.700 | | 25538.500 | ...........................................................................................**X | 25538.500 | | 24713.700 | ........................................................................................X | 24713.700 | | 24458.800 | .......................................................................................****.***..*****.****..****.***X | 26164.100 | | 25218.900 | ..........................................................................................****X | 25218.900 | | 24757.900 | .........................................................................................***.*****.********.********.**X | 25943.000 | | 25691.300 | ............................................................................................X | 25691.300 | | 25645.300 | ............................................................................................***X | 25645.300 | | 25314.300 | ...........................................................................................***X | 25314.300 | | 24858.300 | .........................................................................................*******X | 25021.900 | | 24253.300 | .......................................................................................****X | 24253.300 | | 23923.400 | .....................................................................................***...*****..***********.***X | 25331.200 | | 25143.700 | ..........................................................................................**X | 25143.700 | | 24518.500 | ........................................................................................X | 24518.500 | | 24427.800 | .......................................................................................****.***.*.*******.*******X | 25480.500 | +-----------+--------------------------------------------------------------------------------------------------------------------------------+-----------+ 2023-03-31 04:15:53,845 | test | INFO | pool-3-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..*****...........***................................*......***X | 382.000 | | 302.800 | ........................................***X | 302.800 | | 0.000 | ********....*...****...............****.............................................****X | 496.000 | | 387.100 | ...................................................**X | 387.100 | | 0.000 | *************........****.......................................................................................*****....**X | 738.100 | | 96.500 | ............****X | 96.500 | | 0.000 | ***********........*..****...***X | 100.000 | | 19.800 | .****X | 19.800 | | 0.000 | ***********...............................*****...................***.............****X | 469.100 | | 330.500 | ...........................................**X | 330.500 | | 70.000 | ........X | 70.000 | | 0.000 | *******................*.....****................***.....................*......***X | 479.700 | | 380.700 | ..................................................***X | 380.700 | | 59.900 | .......****X | 59.900 | | 40.100 | ....*X | 40.100 | | 0.000 | *****....***.............................*.........****....**X | 346.400 | | 226.000 | .............................X | 226.000 | | 186.000 | ........................***X | 186.000 | | 120.300 | ...............**X | 120.300 | | 30.000 | ...X | 30.000 | | 0.000 | ********......****...........****.................................****X | 374.600 | | 305.300 | ........................................**X | 305.300 | | 27.700 | ..****X | 27.700 | | 17.800 | .**X | 17.800 | | 0.000 | *****....***.........................................................*****......**X | 500.200 | | 206.500 | ..........................X | 206.500 | | 96.500 | ............***X | 96.500 | | 18.700 | .**X | 18.700 | | 0.000 | *****.***................*......****....***X | 200.000 | | 79.700 | .........***X | 79.700 | | 40.100 | ....***X | 40.100 | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ 2023-03-31 04:15:54,073 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.107.217 | kv | 81.956970634 | 23.36 GiB | 21.01 GiB | 0.0 Byte / 3.50 GiB | 1615686 / 3202787 | 7.2.0-5284-enterprise | | 172.23.107.222 | kv | 81.0999911389 | 23.36 GiB | 20.91 GiB | 0.0 Byte / 3.50 GiB | 1609676 / 3206741 | 7.2.0-5284-enterprise | | 172.23.107.102 | kv | 82.2880479317 | 23.36 GiB | 21.31 GiB | 0.0 Byte / 3.50 GiB | 1604431 / 3206166 | 7.2.0-5284-enterprise | | 172.23.107.99 | kv | 81.2244599715 | 23.36 GiB | 21.02 GiB | 56.69 MiB / 3.50 GiB | 1632150 / 3286972 | 7.2.0-5284-enterprise | | 172.23.107.223 | kv | 82.1689027261 | 23.36 GiB | 20.94 GiB | 0.0 Byte / 0.0 Byte | 1598839 / 3216980 | 7.2.0-5284-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2023-03-31 04:15:56,569 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4902] Validating collection stats and item counts 2023-03-31 04:16:07,296 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4765] Bucket UID mismatch. Expected: 0, Actual: 1 2023-03-31 04:16:07,296 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4919] Bucket manifest UID mismatch! 2023-03-31 04:16:13,486 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4765] Bucket UID mismatch. Expected: 0, Actual: 1 2023-03-31 04:16:13,486 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4919] Bucket manifest UID mismatch! 2023-03-31 04:16:18,642 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4765] Bucket UID mismatch. Expected: 0, Actual: 1 2023-03-31 04:16:18,644 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4919] Bucket manifest UID mismatch! 2023-03-31 04:16:29,631 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.107.217 | kv | 7.46836661378 | 23.36 GiB | 21.08 GiB | 0.0 Byte / 3.50 GiB | 1667700 / 3306000 | 7.2.0-5284-enterprise | | 172.23.107.222 | kv | 6.7353887021 | 23.36 GiB | 21.02 GiB | 0.0 Byte / 3.50 GiB | 1659150 / 3304950 | 7.2.0-5284-enterprise | | 172.23.107.102 | kv | 7.12473494736 | 23.36 GiB | 21.39 GiB | 0.0 Byte / 3.50 GiB | 1656500 / 3311700 | 7.2.0-5284-enterprise | | 172.23.107.99 | kv | 7.57720434981 | 23.36 GiB | 21.09 GiB | 56.69 MiB / 3.50 GiB | 1654800 / 3333300 | 7.2.0-5284-enterprise | | 172.23.107.223 | kv | 7.77210188435 | 23.36 GiB | 21.02 GiB | 0.0 Byte / 0.0 Byte | 1651850 / 3324050 | 7.2.0-5284-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2023-03-31 04:16:30,161 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 100000 | 9.77 GiB | 320.20 MiB | 337.96 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 466.31 MiB | 309.52 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 8140000 | 2.50 GiB | 1.65 GiB | 17.07 GiB | 21.0615601966 | +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ 2023-03-31 04:16:30,163 | test | INFO | MainThread | [collections_base:setup_collection_history_settings:132] Loading initial dedupe data. Total itr:: 3000 2023-03-31 04:16:54,855 | test | INFO | MainThread | [collections_base:mutate_history_retention_data:245] Loading docs for history_retention testing 2023-03-31 04:16:54,857 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5159] Performing scope/collection specific operations 2023-03-31 04:16:54,865 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5249] Done Performing scope/collection specific operations 2023-03-31 04:17:12,625 | test | INFO | pool-3-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.........***.................................................................*..........****................**X | 3030.400 | +-------+---------------------------------------------------------------------------------------------------------------------+----------+ 2023-03-31 04:17:12,762 | test | INFO | pool-3-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.........***........................................................*....................****...............**X | 2261.900 | +-------+---------------------------------------------------------------------------------------------------------------------+----------+ 2023-03-31 04:17:12,766 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2023-03-31 04:17:16,756 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:1058] Doing collection data load during graceful_failover_recovery 2023-03-31 04:17:16,757 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:374] Starting rebalance operation of type : graceful_failover_recovery 2023-03-31 04:17:16,757 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:684] failing over nodes [ip:172.23.107.102 port:8091 ssh_username:root] 2023-03-31 04:17:38,046 | test | INFO | pool-3-thread-4 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 21.1649999619 seconds to complete 2023-03-31 04:17:38,049 | test | INFO | pool-3-thread-4 | [common_lib:sleep:23] Sleep 5 seconds. Reason: Wait after rebalance complete 2023-03-31 04:17:49,088 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:340] Starting a sync data load after failover 2023-03-31 04:17:49,088 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, 3000) 2023-03-31 04:17:49,088 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5159] Performing scope/collection specific operations 2023-03-31 04:17:49,095 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5249] Done Performing scope/collection specific operations 2023-03-31 04:23:27,365 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5159] Performing scope/collection specific operations 2023-03-31 04:23:27,374 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5249] Done Performing scope/collection specific operations 2023-03-31 04:27:00,904 | test | INFO | pool-3-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...............****.......................****..............................................................****X | 1195.900 | | 1070.700 | ........................................................................................**X | 1070.700 | | 484.600 | .......................................****X | 484.600 | | 439.200 | ...................................***.....****X | 496.900 | | 479.700 | .......................................**X | 479.700 | | 395.400 | ................................X | 395.400 | | 357.100 | ............................****.***X | 368.000 | | 336.500 | ...........................**X | 336.500 | | 336.500 | ...........................*X | 336.500 | | 334.300 | ..........................***....X | 377.300 | | 377.300 | ..............................***X | 377.300 | | 332.600 | ..........................**X | 332.600 | | 235.700 | ..................X | 235.700 | | 187.900 | ..............****.****.....****X | 256.600 | | 240.600 | ...................***.....*****...**X | 345.100 | | 270.900 | .....................****X | 270.900 | | 258.500 | ....................***X | 261.000 | | 247.200 | ...................***X | 247.200 | | 198.800 | ...............****.*****....**X | 251.400 | | 197.700 | ...............****X | 197.700 | | 181.500 | ..............***....***X | 238.000 | | 219.700 | .................**X | 219.700 | | 215.100 | ................X | 215.100 | | 214.900 | ................****..***X | 234.800 | | 234.400 | ..................****X | 234.400 | | 229.200 | ..................**X | 229.200 | | 204.300 | ................****X | 204.300 | | 203.800 | ................***....X | 262.400 | | 259.100 | ....................****...****...****X | 332.100 | | 295.300 | .......................**X | 295.300 | | 168.700 | .............****X | 168.700 | | 131.100 | .........X | 131.100 | +----------+--------------------------------------------------------------------------------------------------------------------+----------+ 2023-03-31 04:27:01,184 | test | INFO | pool-3-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | 22996.600 | ...................................................X | 22996.600 | | 12962.600 | ............................****...****...........................................................*****.........**X | 43897.900 | | 38914.200 | .......................................................................................****X | 38914.200 | | 38381.900 | ......................................................................................***........****X | 41723.800 | | 41514.000 | .............................................................................................**X | 41514.000 | | 37111.100 | ...................................................................................X | 37111.100 | | 34970.600 | ..............................................................................***X | 34970.600 | | 33371.200 | ...........................................................................***X | 33371.200 | | 30335.100 | ....................................................................****X | 30335.100 | | 28831.300 | ................................................................**X | 28831.300 | | 26426.400 | ...........................................................********.*******X | 26938.000 | | 26859.700 | ............................................................X | 26859.700 | | 26791.700 | ............................................................***X | 26791.700 | | 26454.800 | ...........................................................***X | 26454.800 | | 24790.800 | .......................................................****X | 24790.800 | | 24216.100 | ......................................................**X | 24216.100 | | 23100.100 | ...................................................****X | 23100.100 | | 22884.800 | ...................................................**X | 22884.800 | | 21647.600 | ................................................****X | 21647.600 | | 21460.400 | ...............................................***..*.****.***X | 23079.800 | | 22847.600 | ...................................................****X | 22847.600 | | 22748.000 | ..................................................**X | 22748.000 | | 22563.700 | ..................................................***X | 22563.700 | | 22417.000 | ..................................................**X | 22417.000 | | 22131.800 | .................................................X | 22131.800 | | 21995.200 | .................................................********.***X | 22652.500 | | 22549.500 | ..................................................***X | 22549.500 | | 21871.700 | ................................................****X | 21871.700 | | 21765.100 | ................................................********.********.***X | 22407.600 | | 22293.000 | .................................................****X | 22293.000 | | 21982.000 | .................................................X | 21982.000 | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ 2023-03-31 04:27:02,266 | test | INFO | pool-3-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *......****...........................****...................................................................****X | 420.900 | | 395.800 | .............................................................................................**X | 395.800 | | 281.500 | .................................................................****X | 281.500 | | 253.900 | ...........................................................**X | 253.900 | | 239.600 | .......................................................*****....**X | 255.000 | | 209.500 | ................................................X | 209.500 | | 208.000 | ................................................*******X | 210.300 | | 146.500 | .................................****X | 146.500 | | 136.700 | ...............................***..................................****X | 281.000 | | 247.900 | .........................................................**X | 247.900 | | 127.300 | .............................X | 127.300 | | 77.500 | .................****....****..................*****..***..............****X | 238.000 | | 202.700 | ...............................................**X | 202.700 | | 64.600 | ..............*****.....*X | 84.500 | | 84.500 | ...................*...........*.***X | 137.100 | | 114.600 | ..........................****.....*****...**X | 147.500 | | 93.800 | .....................*****...***.....***X | 130.200 | | 102.600 | .......................**X | 102.600 | | 94.800 | .....................X | 94.800 | | 91.300 | ....................****.....****................*****..**X | 186.100 | | 114.800 | ..........................****X | 114.800 | | 97.000 | ......................**X | 97.000 | | 96.800 | .....................X | 96.800 | | 85.600 | ...................****.....***.........*......***X | 169.800 | | 159.800 | ....................................***X | 159.800 | | 94.600 | .....................****X | 94.600 | | 60.800 | .............*X | 60.800 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2023-03-31 04:27:02,573 | test | WARNING | MainThread | [rest_client:get_nodes:1782] 172.23.107.102 - Node not part of cluster inactiveFailed 2023-03-31 04:27:04,688 | test | WARNING | MainThread | [rest_client:get_nodes:1782] 172.23.107.102 - Node not part of cluster inactiveFailed 2023-03-31 04:27:06,819 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, -1) 2023-03-31 04:27:06,821 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5159] Performing scope/collection specific operations 2023-03-31 04:27:06,826 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5249] Done Performing scope/collection specific operations 2023-03-31 04:27:06,829 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5159] Performing scope/collection specific operations 2023-03-31 04:27:06,835 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5249] Done Performing scope/collection specific operations 2023-03-31 04:27:07,089 | test | INFO | pool-3-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.107.217 | kv | 7.2.0-5284-enterprise | 68.0534648983 | Cluster node | active / none | | 172.23.107.222 | kv | 7.2.0-5284-enterprise | 76.7772972122 | Cluster node | active / none | | 172.23.107.102 | kv | 7.2.0-5284-enterprise | 1.3411772874 | Cluster node | inactiveAdded / delta | | 172.23.107.99 | kv | 7.2.0-5284-enterprise | 75.0735215964 | Cluster node | active / none | | 172.23.107.223 | kv | 7.2.0-5284-enterprise | 75.620712672 | Cluster node | active / none | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-03-31 04:27:07,171 | test | INFO | pool-3-thread-4 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-03-31 04:27:17,214 | test | INFO | pool-3-thread-4 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-03-31 04:27:27,257 | test | INFO | pool-3-thread-4 | [task:check:421] Rebalance - status: running, progress: 24.21 2023-03-31 04:27:35,217 | test | ERROR | pool-3-thread-6 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_4603_bucket2_scope-7_collection-79_update_ttl=0_update_MAJORITY_196_272_5183 2023-03-31 04:27:37,319 | test | INFO | pool-3-thread-4 | [task:check:421] Rebalance - status: running, progress: 58.21 2023-03-31 04:27:43,108 | test | ERROR | pool-3-thread-6 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 580 to 656 of thread_name LoadDocs_MutateDocsFromSpecTask_4603_bucket2_scope-2_collection-36_create_ttl=0_create_MAJORITY_580_656_5295 2023-03-31 04:27:47,371 | test | ERROR | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1551] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'dff6d03e5a470dbf8e9a4274def2a135', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=2ded308e1e5c978caf346f91c6452f32', u'status': u'notRunning'} - rebalance failed 2023-03-31 04:27:47,421 | test | INFO | pool-3-thread-4 | [rest_client:print_UI_logs:2733] Latest logs from UI on 172.23.107.222: 2023-03-31 04:27:47,423 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262059524L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:39.524Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.9111.9>,\n {{dcp_wait_for_data_move_failed,"default",\n 102,\'ns_1@172.23.107.222\',\n [\'ns_1@172.23.107.102\',\'ns_1@172.23.107.99\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,451}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = ed6c9ae042b68288d93e85b9fff7d6af'} 2023-03-31 04:27:47,424 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262059470L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:39.470Z', u'text': u'Worker <0.8900.9> (for action {move,{102,\n [\'ns_1@172.23.107.222\',\n \'ns_1@172.23.107.99\',\n \'ns_1@172.23.107.102\'],\n [\'ns_1@172.23.107.102\',\n \'ns_1@172.23.107.222\',\n \'ns_1@172.23.107.99\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.9111.9>,\n {{dcp_wait_for_data_move_failed,\n "default",\n 102,\n \'ns_1@172.23.107.222\',\n [\'ns_1@172.23.107.102\',\n \'ns_1@172.23.107.99\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 451}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2023-03-31 04:27:47,427 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262058974L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:38.974Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2023-03-31 04:27:47,427 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262058588L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:38.588Z', u'text': u'Started rebalancing bucket default'} 2023-03-31 04:27:47,427 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262049800L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:29.800Z', u'text': u'Bucket "bucket2" rebalance does not seem to be swap rebalance'} 2023-03-31 04:27:47,427 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262049484L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:29.484Z', u'text': u'Started rebalancing bucket bucket2'} 2023-03-31 04:27:47,428 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262041816L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:21.816Z', u'text': u'Bucket "bucket1" rebalance does not seem to be swap rebalance'} 2023-03-31 04:27:47,428 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.107.222', u'tstamp': 1680262041516L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:21.516Z', u'text': u'Started rebalancing bucket bucket1'} 2023-03-31 04:27:47,430 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.102', u'tstamp': 1680262039912L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:19.912Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.102\' in 4 seconds.'} 2023-03-31 04:27:47,430 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2735] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.107.102', u'tstamp': 1680262034572L, u'shortText': u'message', u'serverTime': u'2023-03-31T04:27:14.572Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.107.102\' in 1 seconds.'} 2023-03-31 04:27:47,430 | test | ERROR | pool-3-thread-4 | [task:call:332] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'dff6d03e5a470dbf8e9a4274def2a135', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=2ded308e1e5c978caf346f91c6452f32', u'status': u'notRunning'} - rebalance failed