2021-04-01 01:02:16,092 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-04-01 01:02:16,569 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-04-01 01:02:16,788 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-04-01 01:04:51,420 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:591] RAM quota was defined less than 100 MB: 2021-04-01 01:05:04,016 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.106.13, 172.23.106.16, 172.23.106.15, 172.23.106.11, 172.23.104.251, 172.23.104.253 initialized 2021-04-01 01:05:04,028 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_out ========= 2021-04-01 01:05:04,043 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-04-01 01:05:14,032 | test | INFO | MainProcess | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.106.13 | kv | 7.0.0-4834-enterprise | 0.376695128076 | Cluster node | | 172.23.106.16 | None | | | <--- IN --- | | 172.23.106.15 | None | | | <--- IN --- | | 172.23.106.11 | None | | | <--- IN --- | | 172.23.104.251 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-04-01 01:05:14,046 | test | INFO | MainProcess | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-01 01:05:24,082 | test | INFO | MainProcess | pool-3-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-01 01:05:24,095 | test | INFO | MainProcess | pool-3-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.0650000572 sec 2021-04-01 01:05:24,233 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-01 01:05:33,221 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-04-01 01:05:33,286 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | default | couchbase | 2 | none | 0 | 0 | 4194304000 | 143352888 | 31851599 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-04-01 01:05:33,288 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-04-01 01:05:49,395 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-04-01 01:05:49,403 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-04-01 01:07:15,999 | test | INFO | MainProcess | pool-3-thread-10 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..***........................**..........................................................***................*X | 85334.377 | | 84957.178 | ..................................................................................................X | 84957.178 | | 84345.747 | .................................................................................................**X | 84345.747 | | 84021.753 | .................................................................................................*X | 84021.753 | | 83374.455 | ................................................................................................*****.**X | 84380.876 | | 83352.500 | ................................................................................................*X | 83352.500 | | 80747.882 | .............................................................................................**X | 80747.882 | | 80054.459 | ............................................................................................*X | 80054.459 | | 79009.194 | ...........................................................................................X | 79009.194 | | 78658.099 | ...........................................................................................*X | 78658.099 | | 74232.023 | .....................................................................................*X | 74232.023 | | 67368.823 | .............................................................................X | 67368.823 | | 66842.139 | .............................................................................*X | 66842.139 | | 58368.939 | ...................................................................**X | 58368.939 | | 57258.400 | ..................................................................*X | 57258.400 | +-----------+------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-01 01:07:16,065 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.106.16 | kv | 99.5489852167 | 4201840640 | 2684788736 | 189005824 / 3758092288 | 924878 / 1862423 | 7.0.0-4834-enterprise | | 172.23.106.13 | kv | 86.8730808598 | 4201840640 | 2730569728 | 0 / 3758092288 | 1194452 / 2400719 | 7.0.0-4834-enterprise | | 172.23.106.15 | kv | 87.5 | 4201840640 | 2739314688 | 0 / 3758092288 | 1186693 / 2390345 | 7.0.0-4834-enterprise | | 172.23.106.11 | kv | 89.5621181263 | 4201844736 | 2832695296 | 227631104 / 3758092288 | 1167353 / 2308436 | 7.0.0-4834-enterprise | | 172.23.104.251 | kv | 89.755351682 | 4219441152 | 2843521024 | 34340864 / 3758092288 | 1160623 / 2297062 | 7.0.0-4834-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-04-01 01:07:23,907 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-04-01 01:07:43,092 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ | default | couchbase | 2 | none | 0 | 6400000 | 4194304000 | 2390737096 | 4158093834 | +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ 2021-04-01 01:07:43,094 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_rebalance_out ========= 2021-04-01 01:07:46,746 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:821] Doing collection data load during rebalance_out 2021-04-01 01:07:46,746 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_to_dgm:225] Loading docs with maxttl:0 in default collection in order to load bucket in dgm 2021-04-01 01:07:51,039 | test | INFO | MainProcess | pool-3-thread-4 | [task:call:2098] Starting DGM doc loading task 2021-04-01 01:07:51,966 | test | INFO | MainProcess | pool-3-thread-4 | [task:_load_bucket_into_dgm:2082] DGM doc loading for 'default' to atleast 55% 2021-04-01 01:07:51,966 | test | INFO | MainProcess | pool-3-thread-4 | [task:_load_bucket_into_dgm:2092] Active DGM 90.1669375% Replica DGM 37.209390625% achieved for 'default'. Loaded docs: 0 2021-04-01 01:07:52,190 | test | INFO | MainProcess | pool-3-thread-4 | [task:call:2115] Done loading docs for DGM 2021-04-01 01:07:55,355 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ | default | couchbase | 2 | none | 0 | 6400000 | 4194304000 | 2390737096 | 4158093834 | +---------+-----------+----------+------------+-----+---------+------------+------------+------------+ 2021-04-01 01:07:55,355 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:302] Starting rebalance operation of type : rebalance_out 2021-04-01 01:07:55,385 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-04-01 01:07:55,461 | test | INFO | MainProcess | pool-3-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.16 | kv | 7.0.0-4834-enterprise | 9.55882352941 | Cluster node | | 172.23.106.13 | kv | 7.0.0-4834-enterprise | 9.50695322377 | Cluster node | | 172.23.106.15 | kv | 7.0.0-4834-enterprise | 6.45079686314 | Cluster node | | 172.23.106.11 | kv | 7.0.0-4834-enterprise | 5.90759909114 | --- OUT ---> | | 172.23.104.251 | kv | 7.0.0-4834-enterprise | 6.08040201005 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-04-01 01:07:55,482 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-01 01:07:58,746 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-04-01 01:08:00,592 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.9 2021-04-01 01:08:05,701 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.41 2021-04-01 01:08:10,832 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.41 2021-04-01 01:08:15,894 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.41 2021-04-01 01:08:18,141 | test | INFO | MainProcess | pool-3-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.*......***..............................**...............................................................X | 73309.105 | | 73309.105 | ..................................................................................................*X | 73309.105 | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-01 01:08:20,964 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 3.36 2021-04-01 01:08:27,203 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 17.67 2021-04-01 01:08:32,242 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 29.79 2021-04-01 01:08:37,276 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.79 2021-04-01 01:08:42,325 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 53.93 2021-04-01 01:08:47,362 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.03 2021-04-01 01:08:52,404 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 77.59 2021-04-01 01:08:57,451 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 88.56 2021-04-01 01:09:02,500 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 99.08 2021-04-01 01:09:07,533 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-04-01 01:09:12,568 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-04-01 01:09:22,615 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-01 01:09:22,680 | test | INFO | MainProcess | pool-3-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 87.2180001736 sec 2021-04-01 01:09:25,265 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-04-01 01:09:35,009 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:92] Bucket: default, Active Resident ratio(DGM): 65% 2021-04-01 01:09:35,009 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:95] Bucket: default, Replica Resident ratio(DGM): 38% 2021-04-01 01:09:46,091 | test | INFO | MainProcess | MainThread | [basetestcase:check_if_new_messages:697] Comparing timestamps: Log's latest timestamp: 2021-04-01 01:08:01, Test's start timestamp is 2021-04-01 01:02:16.093000 2021-04-01 01:09:46,092 | test | INFO | MainProcess | MainThread | [basetestcase:check_coredump_exist:795] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-04-01 01:09:46,092 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:797] 172.23.106.15: Found ' ERROR ' logs - ['2021-04-01T01:08:00.924756-07:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:241 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file\n', '2021-04-01T01:08:01.002835-07:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:245 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file\n'] 2021-04-01 01:09:47,782 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1087] Running cbcollect on node ns_1@172.23.106.13,ns_1@172.23.106.15,ns_1@172.23.106.16 2021-04-01 01:09:57,875 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1090] ns_1@172.23.106.13,ns_1@172.23.106.15,ns_1@172.23.106.16 - cbcollect status: True 2021-04-01 01:09:57,875 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1095] Polling active_tasks to check cbcollect status 2021-04-01 01:12:28,138 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.13: Copying cbcollect ZIP file to Client 2021-04-01 01:12:29,010 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.13 node cb collect zip copied on client : True 2021-04-01 01:12:29,010 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.15: Copying cbcollect ZIP file to Client 2021-04-01 01:12:29,839 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.15 node cb collect zip copied on client : True 2021-04-01 01:12:29,839 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.16: Copying cbcollect ZIP file to Client 2021-04-01 01:12:30,651 | test | ERROR | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.16 node cb collect zip copied on client : True 2021-04-01 01:12:43,081 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.0590000153 seconds to complete 2021-04-01 01:12:43,088 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete