2021-05-16 18:33:54,026 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-16 18:33:55,012 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-16 18:33:55,510 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:320] Initializing cluster 2021-05-16 18:34:00,986 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:03,006 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:06,032 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:08,052 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:10,072 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:12,086 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:14,108 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:21,453 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:23,473 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:25,494 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:27,516 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:29,536 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:31,556 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:33,576 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:40,984 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:43,006 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:45,026 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:47,046 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:49,069 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:51,089 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:53,111 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:34:55,131 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:03,224 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:05,246 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:07,266 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:09,286 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:11,306 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:13,326 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:15,348 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:20,017 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:22,036 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:24,061 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:26,079 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:31,407 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:33,428 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:35,451 | test | ERROR | MainProcess | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.201:8091/nodes/self error [Errno 111] Connection refused 2021-05-16 18:35:52,910 | test | WARNING | MainProcess | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-16 18:36:07,806 | test | INFO | MainProcess | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206, 172.23.120.201 initialized 2021-05-16 18:36:07,834 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-16 18:36:07,875 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-16 18:36:27,618 | test | INFO | MainProcess | pool-21-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5169-enterprise | 8.16944024206 | Cluster node | | 172.23.98.195 | ['fts'] | | | <--- IN --- | | 172.23.121.10 | ['kv'] | | | <--- IN --- | | 172.23.104.186 | ['kv'] | | | <--- IN --- | | 172.23.120.206 | ['kv'] | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-16 18:36:27,648 | test | INFO | MainProcess | pool-21-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-16 18:36:37,740 | test | INFO | MainProcess | pool-21-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-16 18:36:37,770 | test | INFO | MainProcess | pool-21-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.1529998779 sec 2021-05-16 18:36:38,102 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-16 18:37:16,792 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-16 18:37:17,095 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 679030368 | 174158823 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 659613728 | 106310744 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 678991904 | 110784360 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-16 18:37:17,096 | test | INFO | MainProcess | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-16 18:38:01,756 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-16 18:38:01,769 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-16 18:53:07,519 | test | INFO | MainProcess | pool-21-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****....................................................................................................***X | 58.481 | | 42.881 | ........................................................................X | 42.881 | | 0.000 | *********.......****...X | 6.000 | | 1.600 | .***X | 1.600 | | 0.000 | *********.*....***X | 3.000 | | 2.200 | ..X | 2.200 | | 0.000 | *******************************.*....***X | 3.000 | | 2.600 | ...X | 2.600 | | 1.500 | .***....*...........***X | 10.501 | | 8.500 | .............X | 8.500 | | 3.000 | ....***X | 3.000 | | 2.200 | ..X | 2.200 | | 0.000 | ****.*......***X | 4.499 | | 3.400 | ....***X | 3.400 | | 3.000 | ....****X | 3.001 | | 3.000 | ....***X | 3.000 | | 2.200 | ..X | 2.200 | | 0.000 | *****..**X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | *****.****.......*.............**X | 12.400 | | 9.200 | ..............X | 9.200 | | 1.500 | .****.*....***X | 4.499 | | 0.000 | *******************...****..*.***X | 3.801 | | 2.601 | ...X | 2.601 | | 0.000 | *****.......***...X | 6.001 | | 1.600 | .***X | 1.600 | | 0.800 | *....***X | 3.001 | | 2.601 | ...X | 2.601 | | 1.500 | .***...*.......****...*.........**X | 14.603 | | 11.800 | ...................*X | 11.800 | | 1.900 | ..****..***X | 3.000 | | 2.200 | ..X | 2.200 | | 0.000 | ********..*........**X | 6.000 | | 4.400 | ......X | 4.400 | | 0.000 | *************......**.................**X | 13.502 | | 9.902 | ...............X | 9.902 | | 0.000 | ****.......****.*X | 4.901 | | 4.900 | .......***...*.........***X | 11.997 | | 9.197 | ..............X | 9.197 | | 1.500 | .***X | 1.500 | | 0.000 | *****..***X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ****.................***X | 10.499 | | 7.699 | ............*X | 7.699 | | 0.000 | ********..****..*..........*******X | 8.304 | | 5.500 | ........X | 5.500 | | 0.000 | ****..*........**X | 6.002 | | 4.800 | .......*X | 4.800 | | 1.500 | .**X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ****....****.X | 2.999 | | 1.900 | ..***X | 1.900 | | 1.100 | X | 1.100 | | 0.000 | ***..**........**X | 6.000 | | 4.400 | ......X | 4.400 | | 1.800 | ..***......*...........***X | 11.699 | | 8.099 | ............X | 8.099 | | 0.000 | ****......*...................**X | 15.000 | | 11.400 | ..................*X | 11.400 | | 1.500 | .**X | 1.500 | | 1.499 | .*****.*..**X | 3.001 | | 2.200 | ..X | 2.200 | | 0.000 | ****.*...........****...X | 9.002 | | 4.600 | ......***X | 4.600 | | 2.200 | ..X | 2.200 | | 0.000 | *******************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------+--------+ 2021-05-16 18:53:09,451 | test | INFO | MainProcess | pool-21-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****..........................*..........................................................................**X | 40.487 | | 29.687 | ........................................................................*X | 29.687 | | 0.000 | *********........***X | 3.300 | | 1.200 | .***X | 1.200 | | 0.400 | *......****....*.....***X | 6.401 | | 6.000 | .............***X | 6.000 | | 4.400 | .........X | 4.400 | | 0.000 | ****.*......*******X | 3.000 | | 2.200 | ....X | 2.200 | | 0.000 | ******************..*.........**X | 4.500 | | 4.100 | .........X | 4.100 | | 3.000 | ......****....*...................****.X | 12.700 | | 7.200 | ................**X | 7.200 | | 4.400 | .........X | 4.400 | | 0.000 | ****...*...................****.X | 9.698 | | 4.200 | .........**X | 4.200 | | 3.400 | .......*...***X | 4.501 | | 3.701 | ........X | 3.701 | | 1.500 | ..***X | 1.500 | | 1.100 | .X | 1.100 | | 0.000 | ****.*......**X | 3.001 | | 2.199 | ....X | 2.199 | | 0.000 | ********.*......***X | 3.000 | | 2.600 | .....X | 2.600 | | 1.500 | ..***X | 1.500 | | 1.500 | ..***X | 1.500 | | 1.499 | ..*********X | 1.500 | | 1.100 | .X | 1.100 | | 0.000 | ********.*......****..*.....***X | 6.001 | | 4.801 | ..........X | 4.801 | | 1.500 | ..**X | 1.500 | | 1.100 | .*...........****.....X | 7.501 | | 2.000 | ...***X | 2.000 | | 1.500 | ..*****X | 1.500 | | 1.500 | ..***...*........****.........................***X | 16.103 | | 12.900 | ..............................X | 12.900 | | 1.900 | ...***X | 1.900 | | 1.500 | ..***X | 1.500 | | 1.100 | .*X | 1.100 | | 0.000 | ********..................***X | 7.499 | | 5.499 | ............X | 5.499 | | 0.000 | *************......*...................***X | 10.501 | | 0.000 | ****..**.........**X | 4.500 | | 4.101 | .........X | 4.101 | | 3.000 | ......****.............***X | 8.198 | | 7.798 | ..................*X | 7.798 | | 2.300 | ....**X | 2.300 | | 1.500 | ..****X | 1.500 | | 1.100 | .X | 1.100 | | 0.000 | ********..**.........**X | 4.500 | | 3.300 | .......X | 3.300 | | 0.000 | **********...***...........*......................................***X | 21.297 | | 7.204 | ................***X | 7.204 | | 4.400 | .........X | 4.400 | | 0.000 | *****...***..**......**X | 4.499 | | 3.299 | .......X | 3.299 | | 0.000 | ****.....****..X | 2.999 | | 0.800 | ***X | 0.800 | | 0.000 | ****....**..............**X | 7.500 | | 5.500 | ............X | 5.500 | | 4.000 | ........****......X | 6.400 | | 5.000 | ...........**X | 5.000 | | 2.600 | .....X | 2.600 | | 0.000 | ****........*.........................***X | 13.500 | | 10.700 | .........................X | 10.700 | | 3.000 | ......**X | 3.000 | | 2.200 | ....X | 2.200 | | 0.000 | *****...****.*...****..........****.....X | 9.002 | | 3.500 | .......***X | 3.500 | | 1.100 | .X | 1.100 | | 0.000 | ********************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------+--------+ 2021-05-16 18:53:10,947 | test | INFO | MainProcess | pool-21-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****....................................................................................................***X | 1219.819 | | 1215.883 | ..................................................................................................X | 1215.883 | | 1009.458 | .................................................................................***X | 1009.458 | | 975.214 | ..............................................................................X | 975.214 | | 968.670 | ..............................................................................***.*......***X | 1051.253 | | 1004.653 | .................................................................................X | 1004.653 | | 817.500 | ..................................................................***X | 817.500 | | 775.234 | ..............................................................X | 775.234 | | 615.234 | .................................................**X | 615.234 | | 574.449 | ..............................................X | 574.449 | | 497.349 | .......................................****X | 498.700 | | 498.200 | .......................................****....*...........***X | 681.400 | | 657.800 | ....................................................***X | 657.800 | | 642.100 | ...................................................****X | 642.300 | | 624.000 | ..................................................X | 624.000 | | 508.400 | ........................................***X | 508.400 | | 487.400 | ......................................*.***.X | 508.200 | | 475.136 | .....................................***X | 475.136 | | 467.636 | .....................................*.......****.....*......**X | 686.400 | | 659.469 | .....................................................X | 659.469 | | 621.269 | .................................................***X | 621.269 | | 601.227 | ................................................X | 601.227 | | 526.027 | ..........................................**X | 526.027 | | 493.400 | .......................................X | 493.400 | | 419.600 | .................................****.*..***X | 451.435 | | 433.640 | ..................................X | 433.640 | | 419.976 | .................................***.....*..............***X | 651.268 | | 644.168 | ...................................................X | 644.168 | | 624.026 | ..................................................*****.**X | 637.300 | | 611.829 | .................................................X | 611.829 | | 503.629 | ........................................***X | 503.629 | | 488.251 | .......................................*.***...*....***X | 587.369 | | 554.884 | ............................................X | 554.884 | | 530.751 | ..........................................***.*...****..*.**X | 616.900 | | 599.000 | ................................................X | 599.000 | | 555.538 | ............................................***X | 555.538 | | 524.094 | .........................................X | 524.094 | | 390.798 | ...............................*****......*****...***.X | 512.700 | | 508.200 | ........................................****..*..*****.**X | 569.273 | | 543.113 | ...........................................X | 543.113 | | 494.342 | .......................................***X | 494.342 | | 491.261 | .......................................X | 491.261 | | 473.121 | .....................................**X | 473.121 | | 463.062 | ....................................X | 463.062 | | 390.736 | ...............................***X | 390.736 | | 388.236 | ..............................*.......****....****..*........***X | 641.514 | | 579.198 | ..............................................X | 579.198 | | 459.970 | ....................................****..***X | 486.900 | | 439.300 | ...................................X | 439.300 | | 375.700 | .............................****.*...****....*****....****..*....****.****...*....***X | 695.116 | | 615.216 | .................................................X | 615.216 | | 429.756 | ..................................***..**..**X | 480.262 | | 467.018 | .....................................X | 467.018 | | 462.189 | ....................................****.X | 466.380 | | 433.492 | ..................................***.**.......***...*.....***X | 629.050 | | 591.475 | ...............................................X | 591.475 | | 503.992 | ........................................**X | 503.992 | | 488.833 | .......................................*X | 488.833 | | 435.766 | ..................................**X | 435.766 | | 431.000 | ..................................*.****.X | 456.956 | | 438.382 | ..................................****....*..............***..X | 675.301 | | 669.702 | .....................................................***X | 669.702 | | 624.700 | ..................................................***X | 624.700 | | 605.417 | ................................................X | 605.417 | | 454.285 | ....................................***X | 454.285 | | 414.093 | ................................X | 414.093 | | 388.926 | ..............................***..**....***.X | 466.323 | | 460.723 | ....................................***X | 460.723 | | 445.700 | ...................................X | 445.700 | | 425.501 | .................................***...*...........****..X | 610.238 | | 580.637 | ..............................................*****..**X | 606.005 | | 563.109 | .............................................X | 563.109 | | 411.443 | ................................*****...***X | 440.000 | | 435.100 | ..................................X | 435.100 | | 423.900 | .................................**X | 423.900 | | 411.400 | ................................*X | 411.400 | | 364.464 | ............................***....*.............***X | 567.560 | | 559.800 | ............................................***X | 559.800 | | 541.100 | ...........................................X | 541.100 | | 482.800 | ......................................****.*.**X | 508.316 | | 484.551 | ......................................X | 484.551 | | 426.451 | .................................***X | 426.451 | | 424.900 | .................................*...****.X | 470.301 | | 424.874 | .................................**X | 424.874 | | 410.774 | ................................*....****...*....****.........****.X | 663.405 | | 589.276 | ...............................................***X | 589.276 | | 567.849 | .............................................*.***.......*.............***X | 818.969 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-16 18:53:11,180 | test | INFO | MainProcess | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 43.9727463312 | 3693387776 | 2432696320 | 0 / 3909087232 | 125700 / 253317 | 7.0.0-5169-enterprise | | 172.23.98.195 | fts | 3.44480764395 | 3693387776 | 3054747648 | 0 / 3909087232 | 0 / 0 | 7.0.0-5169-enterprise | | 172.23.104.186 | kv | 53.543512246 | 6078930944 | 4789002240 | 0 / 3758092288 | 125950 / 245251 | 7.0.0-5169-enterprise | | 172.23.120.206 | kv | 27.8095733611 | 3691520000 | 2680926208 | 0 / 0 | 125700 / 252825 | 7.0.0-5169-enterprise | | 172.23.121.10 | kv | 53.8680465718 | 4201836544 | 3074011136 | 0 / 3758092288 | 125098 / 253317 | 7.0.0-5169-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-16 18:53:22,273 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-16 18:54:48,743 | test | INFO | MainProcess | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155100624 | 233247954 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155063040 | 252545183 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 436562192 | 460702153 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-16 18:54:48,743 | test | INFO | MainProcess | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-16 18:54:57,023 | test | INFO | MainProcess | MainThread | [collections_rebalance:setUp:78] Creating metakv entries start 2021-05-16 18:56:05,686 | test | INFO | MainProcess | MainThread | [collections_rebalance:setUp:80] Creating metakv entries end 2021-05-16 18:56:05,687 | test | INFO | MainProcess | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-16 18:56:05,687 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-16 18:56:05,687 | test | INFO | MainProcess | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.120.206 port:8091 ssh_username:root] 2021-05-16 18:56:54,904 | test | INFO | MainProcess | pool-21-thread-26 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 48.9579999447 seconds to complete 2021-05-16 18:56:54,911 | test | INFO | MainProcess | pool-21-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-16 18:58:59,993 | test | INFO | MainProcess | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0770001411438 seconds 2021-05-16 18:58:59,994 | test | INFO | MainProcess | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-16 18:58:59,996 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-16 18:59:05,951 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-16 19:11:25,273 | test | INFO | MainProcess | pool-21-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.*..****...............****...*.............****...........*.............***.**.......*X | 173.700 | | 147.800 | .......................................................*...***.............X | 188.965 | | 147.641 | .......................................................***X | 147.641 | | 111.293 | .........................................*.....***X | 123.151 | | 121.500 | .............................................****......................*................................***X | 259.600 | | 196.200 | ..........................................................................X | 196.200 | | 163.100 | .............................................................***................X | 204.100 | | 178.400 | ...................................................................***X | 178.400 | | 143.500 | ......................................................X | 143.500 | | 115.284 | ...........................................****X | 116.484 | | 105.216 | .......................................**X | 105.216 | | 103.716 | ......................................*........***X | 123.500 | | 116.780 | ...........................................*.....***......X | 143.929 | | 121.111 | .............................................***X | 121.111 | | 119.882 | .............................................*........................****...........X | 211.929 | | 177.400 | ...................................................................***X | 177.400 | | 58.076 | .....................****..*.....****..X | 82.750 | | 68.919 | .........................**X | 68.919 | | 62.100 | ......................*X | 62.100 | | 48.500 | .................**X | 48.500 | | 41.700 | ...............*.***X | 46.600 | | 46.400 | ................*.***X | 47.182 | | 34.309 | ............****..*.......***X | 58.700 | | 44.900 | ................X | 44.900 | | 23.300 | .......***X | 23.300 | | 1.100 | ********X | 1.799 | | 1.701 | X | 1.701 | | 1.701 | ***X | 1.701 | | 0.300 | ********X | 1.400 | | 0.700 | X | 0.700 | | 0.400 | *******X | 1.200 | | 0.800 | X | 0.800 | | 0.500 | ********X | 1.101 | | 0.700 | *****.****X | 2.900 | | 2.400 | **X | 2.400 | | 1.799 | X | 1.799 | | 1.300 | *****.*******X | 4.001 | | 2.701 | X | 2.701 | | 0.800 | *****.***X | 2.699 | | 1.699 | X | 1.699 | | 0.600 | *********.**X | 2.600 | | 2.400 | *.****X | 3.898 | | 2.701 | ***X | 2.701 | | 0.400 | ********X | 2.399 | | 1.699 | X | 1.699 | | 1.500 | ********.****X | 4.100 | | 2.499 | ***X | 2.499 | | 2.499 | *.***X | 2.701 | | 2.401 | X | 2.401 | | 1.500 | ***X | 1.500 | | 1.199 | X | 1.199 | | 1.100 | ****.***X | 4.099 | | 3.201 | X | 3.201 | | 1.900 | ****.***X | 4.001 | | 3.401 | X | 3.401 | | 2.800 | ***X | 2.800 | | 2.500 | X | 2.500 | | 2.199 | **X | 2.199 | | 2.099 | X | 2.099 | | 1.701 | ********X | 2.499 | | 2.399 | X | 2.399 | | 2.200 | ***.*****X | 3.001 | | 2.900 | ***X | 2.900 | | 2.399 | X | 2.399 | | 1.400 | *******X | 1.999 | | 1.600 | ****.****X | 3.500 | | 2.699 | X | 2.699 | | 1.800 | X | 1.800 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-16 19:11:26,894 | test | INFO | MainProcess | pool-21-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.*.****....*.....***.....*.............****..............................................***X | 132.200 | | 123.200 | .....................................................................X | 123.200 | | 121.700 | ....................................................................*X | 121.700 | | 94.100 | ....................................................X | 94.100 | | 60.100 | .................................****............................*...........................**X | 157.838 | | 134.991 | ...........................................................................X | 134.991 | | 126.953 | .......................................................................****.......***X | 139.200 | | 138.000 | .............................................................................*X | 138.000 | | 110.900 | ..............................................................****....***...........**......................**X | 175.500 | | 169.200 | ...............................................................................................X | 169.200 | | 156.982 | ........................................................................................***X | 156.982 | | 127.216 | .......................................................................****........*...***X | 147.400 | | 119.884 | ...................................................................X | 119.884 | | 119.684 | ...................................................................****.***X | 121.117 | | 107.284 | ............................................................**........**X | 121.416 | | 114.416 | ................................................................X | 114.416 | | 113.600 | ...............................................................****..****...*.............................***X | 173.728 | | 160.495 | ..........................................................................................X | 160.495 | | 117.529 | .................................................................***X | 117.529 | | 117.200 | .................................................................***X | 117.200 | | 109.400 | .............................................................X | 109.400 | | 80.700 | ............................................****..........*.............***X | 120.760 | | 68.017 | .....................................***X | 68.017 | | 67.517 | .....................................**..............**X | 92.800 | | 79.800 | ............................................X | 79.800 | | 32.600 | .................*****.....****................***X | 69.771 | | 68.440 | .....................................*X | 68.440 | | 54.533 | ..............................**X | 54.533 | | 54.200 | .............................*.......****.....X | 74.686 | | 71.416 | .......................................****.***X | 73.100 | | 72.500 | ........................................X | 72.500 | | 55.600 | ..............................****....*......***X | 72.942 | | 46.302 | .........................X | 46.302 | | 8.902 | ....**X | 8.902 | | 8.200 | ...X | 8.200 | | 0.600 | *****.****..***X | 5.499 | | 3.798 | .X | 3.798 | | 2.500 | ***X | 2.500 | | 2.300 | *****..***X | 5.301 | | 3.301 | X | 3.301 | | 0.600 | ***.*.***X | 4.099 | | 3.199 | X | 3.199 | | 1.400 | ****.*****.***X | 4.500 | | 3.899 | .***X | 3.899 | | 2.398 | X | 2.398 | | 0.500 | ***X | 0.500 | | 0.300 | X | 0.300 | | 0.200 | ****.****.***X | 4.200 | | 3.400 | X | 3.400 | | 2.301 | ****.*.***X | 6.500 | | 5.100 | .X | 5.100 | | 2.199 | **X | 2.199 | | 2.099 | *****.X | 4.502 | | 3.999 | .***X | 3.999 | | 0.900 | ****.*.***X | 4.199 | | 3.201 | X | 3.201 | | 1.600 | ****.*.***X | 4.701 | | 3.701 | .X | 3.701 | | 2.400 | **X | 2.400 | | 1.900 | X | 1.900 | | 1.199 | ****.******.**X | 3.600 | | 3.000 | X | 3.000 | | 2.300 | ***X | 2.300 | | 2.200 | ********X | 3.300 | | 2.899 | X | 2.899 | | 2.201 | *******X | 2.399 | | 1.999 | X | 1.999 | | 1.800 | ****.*.**X | 5.500 | | 4.299 | .X | 4.299 | | 2.900 | *X | 2.900 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-16 19:11:28,536 | test | INFO | MainProcess | pool-21-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...........*.....................***X | 288.056 | | 268.256 | .............................X | 268.256 | | 224.548 | ........................****........................*...........................................**X | 815.884 | | 688.584 | ............................................................................X | 688.584 | | 489.300 | ......................................................***X | 489.300 | | 433.800 | ................................................X | 433.800 | | 331.300 | ....................................**.....*.........................****.................................***X | 884.092 | | 715.935 | ...............................................................................X | 715.935 | | 350.851 | ......................................***X | 350.851 | | 319.472 | ...................................*.................***.................**.**X | 635.400 | | 437.600 | ................................................X | 437.600 | | 206.200 | ......................**X | 206.200 | | 158.200 | ................*X | 158.200 | | 118.100 | ............***..*......****..........*.....**X | 318.549 | | 269.849 | .............................*X | 269.849 | | 256.700 | ............................**X | 256.700 | | 239.958 | ..........................X | 239.958 | | 199.558 | .....................***X | 199.558 | | 160.618 | .................X | 160.618 | | 101.307 | ..........***.....**........**X | 214.329 | | 202.029 | .....................X | 202.029 | | 166.900 | .................****....*...........**X | 294.990 | | 285.130 | ...............................X | 285.130 | | 229.920 | .........................****..**.....**X | 299.376 | | 265.100 | ............................X | 265.100 | | 221.500 | ........................****..X | 243.300 | | 235.200 | .........................***X | 235.200 | | 214.430 | .......................****.*X | 225.130 | | 215.770 | .......................**X | 215.770 | | 214.970 | .......................*......****..X | 288.000 | | 270.100 | .............................***.....*............***X | 416.960 | | 416.639 | ..............................................X | 416.639 | | 361.615 | .......................................***X | 361.615 | | 311.476 | ..................................X | 311.476 | | 254.541 | ...........................****..****........*.....***X | 387.883 | | 299.583 | ................................X | 299.583 | | 204.900 | ......................***X | 204.900 | | 166.700 | .................****....*......****..*........X | 336.481 | | 336.481 | .....................................**......*...............***X | 522.787 | | 504.087 | ........................................................X | 504.087 | | 487.500 | ......................................................****.....................***X | 673.837 | | 619.165 | .....................................................................X | 619.165 | | 521.981 | ..........................................................****....***X | 562.478 | | 528.513 | ..........................................................X | 528.513 | | 502.479 | .......................................................X | 502.479 | | 502.479 | .......................................................**X | 502.479 | | 457.013 | ..................................................X | 457.013 | | 376.567 | .........................................***....*..***X | 428.234 | | 388.834 | ..........................................X | 388.834 | | 381.367 | ..........................................****.................................***X | 680.100 | | 654.140 | ........................................................................X | 654.140 | | 507.340 | ........................................................***X | 507.340 | | 461.259 | ...................................................***X | 461.259 | | 455.315 | ..................................................X | 455.315 | | 380.643 | ..........................................***X | 380.643 | | 368.143 | ........................................*.**X | 374.000 | | 371.700 | .........................................*.****X | 387.578 | | 385.136 | ..........................................****.......*...........***..*..***X | 580.300 | | 557.500 | ..............................................................*.***X | 570.169 | | 543.638 | ............................................................X | 543.638 | | 392.455 | ...........................................***X | 392.455 | | 386.517 | ..........................................*......**X | 439.500 | | 408.069 | .............................................X | 408.069 | | 392.419 | ...........................................****..X | 414.906 | | 408.176 | .............................................****........*..............**X | 604.940 | | 570.568 | ...............................................................X | 570.568 | | 529.290 | ..........................................................***X | 529.290 | | 526.190 | ..........................................................*.***X | 533.135 | | 506.835 | ........................................................X | 506.835 | | 419.297 | ..............................................**X | 419.297 | | 397.797 | ...........................................X | 397.797 | | 396.714 | ...........................................****........***X | 460.274 | | 457.074 | ..................................................*........****.......*..**X | 604.390 | | 603.310 | ...................................................................*........****.X | 683.918 | | 671.672 | ..........................................................................***X | 671.672 | | 626.172 | .....................................................................X | 626.172 | | 496.176 | .......................................................****....***X | 537.790 | | 512.397 | ........................................................*...***...........*............**X | 734.222 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-16 19:11:28,815 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1718] 172.23.120.206 - Node not part of cluster inactiveFailed 2021-05-16 19:11:28,875 | test | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1718] 172.23.120.206 - Node not part of cluster inactiveFailed 2021-05-16 19:11:39,904 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-16 19:11:40,121 | test | INFO | MainProcess | pool-21-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5169-enterprise | 34.0943683409 | Cluster node | | 172.23.98.195 | fts | 7.0.0-5169-enterprise | 2.69047020367 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5169-enterprise | 31.4781720705 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5169-enterprise | 1.15317122086 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5169-enterprise | 20.2247191011 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-16 19:11:40,188 | test | INFO | MainProcess | pool-21-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-16 19:11:45,342 | test | INFO | MainProcess | pool-21-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-16 19:11:46,263 | test | INFO | MainProcess | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-16 19:11:50,415 | test | INFO | MainProcess | pool-21-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-16 19:11:55,500 | test | INFO | MainProcess | pool-21-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-16 19:12:00,589 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'509edf5c2062c203176c4f3d90935e76', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7b98dee29e022ff8f40720c25d29f840', u'status': u'notRunning'} - rebalance failed 2021-05-16 19:12:00,719 | test | INFO | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2656] Latest logs from UI on 172.23.98.196: 2021-05-16 19:12:00,720 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1621217516764L, u'shortText': u'message', u'serverTime': u'2021-05-16T19:11:56.764Z', u'text': u"Rebalance exited with reason {buckets_cleanup_failed,['ns_1@172.23.104.186']}.\nRebalance Operation Id = a82946326a52b2caf90d02ce4d7fa443"} 2021-05-16 19:12:00,720 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1621217516761L, u'shortText': u'message', u'serverTime': u'2021-05-16T19:11:56.761Z', u'text': u"Failed to cleanup old buckets on node 'ns_1@172.23.104.186': {badrpc,\n {'EXIT',\n timeout}}"} 2021-05-16 19:12:00,720 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.10', u'tstamp': 1621217501565L, u'shortText': u'node down', u'serverTime': u'2021-05-16T19:11:41.565Z', u'text': u"Node 'ns_1@172.23.121.10' saw that node 'ns_1@172.23.120.206' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-05-16 19:12:00,721 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.98.195', u'tstamp': 1621217501564L, u'shortText': u'node down', u'serverTime': u'2021-05-16T19:11:41.564Z', u'text': u"Node 'ns_1@172.23.98.195' saw that node 'ns_1@172.23.120.206' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-05-16 19:12:00,721 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.98.196', u'tstamp': 1621217501564L, u'shortText': u'node down', u'serverTime': u'2021-05-16T19:11:41.564Z', u'text': u"Node 'ns_1@172.23.98.196' saw that node 'ns_1@172.23.120.206' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-05-16 19:12:00,723 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.104.186', u'tstamp': 1621217501563L, u'shortText': u'node down', u'serverTime': u'2021-05-16T19:11:41.563Z', u'text': u"Node 'ns_1@172.23.104.186' saw that node 'ns_1@172.23.120.206' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-05-16 19:12:00,723 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 1, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.120.206', u'tstamp': 1621217501426L, u'shortText': u'message', u'serverTime': u'2021-05-16T19:11:41.426Z', u'text': u"Node 'ns_1@172.23.120.206' is leaving cluster."} 2021-05-16 19:12:00,723 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1621217500053L, u'shortText': u'message', u'serverTime': u'2021-05-16T19:11:40.053Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.98.196','ns_1@172.23.98.195',\n 'ns_1@172.23.104.186','ns_1@172.23.121.10'], EjectNodes = [], Failed over and being ejected nodes = ['ns_1@172.23.120.206']; no delta recovery nodes; Operation Id = a82946326a52b2caf90d02ce4d7fa443"} 2021-05-16 19:12:00,723 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1621216613819L, u'shortText': u'message', u'serverTime': u'2021-05-16T18:56:53.819Z', u'text': u'Graceful failover completed successfully.\nRebalance Operation Id = 01c1ff658503f88f3ce87a4425af4c73'} 2021-05-16 19:12:00,724 | test | ERROR | MainProcess | pool-21-thread-27 | [rest_client:print_UI_logs:2658] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1621216613636L, u'shortText': u'message', u'serverTime': u'2021-05-16T18:56:53.636Z', u'text': u"Deactivating failed over nodes ['ns_1@172.23.120.206']"} 2021-05-16 19:12:00,726 | test | ERROR | MainProcess | pool-21-thread-27 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'509edf5c2062c203176c4f3d90935e76', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7b98dee29e022ff8f40720c25d29f840', u'status': u'notRunning'} - rebalance failed 2021-05-16 19:12:02,022 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-16 19:12:02,023 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-16 19:12:22,461 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-16 19:12:22,463 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-16 19:12:42,549 | test | INFO | MainProcess | pool-21-thread-20 | [table_view:display:72] Keys failed in default:l-G%28Er2WDweID2SM-38-202000:Qax6ZwZtRPr-nQG9K-38-203000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-629 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-636 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-505 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-549 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-545 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-586 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-642 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-660 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-16 19:12:47,915 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-16 19:12:47,915 | test | INFO | MainProcess | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-16 19:13:02,885 | test | INFO | MainProcess | pool-21-thread-13 | [table_view:display:72] Keys failed in bucket2:_default:t7n9_JlLqt7QSpGMV-38-165000 +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5 | {'error': 'com.couchbase.client.core.error.UnambiguousTimeoutException | GetRequest, Reason: TIMEOUT | 172.23.121.10:11210 | reason:TIMEOUT | retryAttempts:64 | retryReasons:[KV_COLLECTION_OUTDATED]', 'cas': 0, 'value': {}} | +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-16 19:13:02,887 | test | INFO | MainProcess | pool-21-thread-13 | [table_view:display:72] Keys failed in bucket2:_default:t7n9_JlLqt7QSpGMV-38-165000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-16 19:13:12,611 | infra | ERROR | MainProcess | pool-21-thread-16 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.98.196:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-05-16 19:13:12,617 | infra | ERROR | MainProcess | pool-21-thread-28 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.98.196:8091/pools/default/buckets/bucket1/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-05-16 19:13:17,608 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:68] Pool did not terminate 2021-05-16 19:13:18,611 | infra | ERROR | MainProcess | pool-21-thread-16 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.98.196:8091/pools/default/buckets/default/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-05-16 19:13:18,611 | infra | ERROR | MainProcess | pool-21-thread-28 | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.98.196:8091/pools/default/buckets/bucket1/stats?zoom=minute. Error [Errno -1] Unmapped exception: java.lang.InterruptedException 2021-05-16 19:13:19,611 | infra | ERROR | MainProcess | MainThread | [task_manager:shutdown:68] Pool did not terminate 2021-05-16 19:13:40,216 | test | CRITICAL | MainProcess | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-16 19:13:40,348 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-05-16 19:13:50,404 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.104.186,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-05-16 19:13:50,404 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-05-16 19:21:22,611 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-05-16 19:21:26,732 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-05-16 19:21:30,704 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-05-16 19:21:33,953 | test | INFO | MainProcess | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-05-16 19:21:38,828 | test | CRITICAL | MainProcess | MainThread | [basetestcase:tearDownEverything:491] Skipping get_trace !! 2021-05-16 19:22:05,346 | test | INFO | MainProcess | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 20.3239998817 seconds to complete 2021-05-16 19:22:05,371 | test | INFO | MainProcess | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete