guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.120343.ini GROUP=stop_start_rebalance,rerun=False,disk_optimized_thread_settings=True,get-cbcollect-info=True,autoCompactionDefined=true,upgrade_version=7.2.0-5318 -t rebalance_new.rebalance_start_stop.RebalanceStartStopTests.test_start_stop_rebalance_with_mutations,doc_size=512,upgrade_version=7.2.0-5318,nodes_in=1,rerun=False,GROUP=stop_start_rebalance,nodes_out=0,bucket_history_retention_seconds=60,bucket_spec=magma_dgm.10_percent_dgm.4_node_1_replica_magma_512,dedupe_update_itrs=10000,bucket_history_retention_bytes=100000000000,get-cbcollect-info=True,extra_nodes_out=0,nodes_init=4,default_history_retention_for_collections=false,disk_optimized_thread_settings=True,autoCompactionDefined=true,extra_nodes_in=1,randomize_value=True' Test Input params: {u'doc_size': u'512', 'conf_file': 'conf/magma/history_retention_rebalances.conf', u'upgrade_version': '7.2.0-5318', u'nodes_in': u'1', u'rerun': 'False', 'spec': 'history_retention_rebalances', 'num_nodes': 6, u'GROUP': 'stop_start_rebalance', u'nodes_out': u'0', u'bucket_history_retention_seconds': u'60', u'bucket_spec': u'magma_dgm.10_percent_dgm.4_node_1_replica_magma_512', 'case_number': 3, u'dedupe_update_itrs': u'10000', 'cluster_name': 'testexec.120343', u'bucket_history_retention_bytes': u'100000000000', u'get-cbcollect-info': 'True', 'ini': '/tmp/testexec.120343.ini', 'logs_folder': '/data/workspace/debian-p0-magma-vset00-00-nsserv_collection_reb_start_stop_10DGM_hist_retention_7.2_P0/logs/testrunner-23-Apr-27_21-19-33/test_3', u'extra_nodes_out': u'0', u'nodes_init': u'4', u'default_history_retention_for_collections': u'false', u'disk_optimized_thread_settings': 'True', u'autoCompactionDefined': 'true', u'extra_nodes_in': u'1', u'randomize_value': u'True'} test_start_stop_rebalance_with_mutations (rebalance_new.rebalance_start_stop.RebalanceStartStopTests) ... 2023-04-27 22:30:06,973 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= RebalanceStartStopTests setup started for test #3 test_start_stop_rebalance_with_mutations ========= 2023-04-27 22:30:07,111 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= BaseTestCase setup started for test #3 test_start_stop_rebalance_with_mutations ========= 2023-04-27 22:30:28,857 | test | INFO | MainThread | [rest_client:monitorRebalance:1489] Rebalance done. Taken 11.0990002155 seconds to complete 2023-04-27 22:30:28,865 | test | INFO | MainThread | [common_lib:sleep:23] Sleep 5 seconds. Reason: Wait after rebalance complete 2023-04-27 22:30:34,051 | test | INFO | MainThread | [basetestcase:initialize_cluster:463] Initializing cluster : C1 2023-04-27 22:30:36,446 | test | ERROR | pool-8-thread-16 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.163:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:36,479 | test | ERROR | pool-8-thread-15 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.141:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:36,655 | test | ERROR | pool-8-thread-17 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.161:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:37,098 | test | ERROR | pool-8-thread-14 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.139:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:37,117 | test | ERROR | pool-8-thread-13 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.150:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:37,197 | test | ERROR | pool-8-thread-18 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.162:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:38,454 | test | ERROR | pool-8-thread-16 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.163:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:38,484 | test | ERROR | pool-8-thread-15 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.141:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:38,661 | test | ERROR | pool-8-thread-17 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.161:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:39,105 | test | ERROR | pool-8-thread-14 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.139:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:39,121 | test | ERROR | pool-8-thread-13 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.150:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:39,203 | test | ERROR | pool-8-thread-18 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.162:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:40,460 | test | ERROR | pool-8-thread-16 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.163:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:40,490 | test | ERROR | pool-8-thread-15 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.141:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:40,667 | test | ERROR | pool-8-thread-17 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.161:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:41,109 | test | ERROR | pool-8-thread-14 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.139:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:41,127 | test | ERROR | pool-8-thread-13 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.150:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:41,207 | test | ERROR | pool-8-thread-18 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.162:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:42,467 | test | ERROR | pool-8-thread-16 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.163:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:42,494 | test | ERROR | pool-8-thread-15 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.141:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:42,673 | test | ERROR | pool-8-thread-17 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.161:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:43,115 | test | ERROR | pool-8-thread-14 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.139:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:43,131 | test | ERROR | pool-8-thread-13 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.150:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:43,213 | test | ERROR | pool-8-thread-18 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.162:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:44,471 | test | ERROR | pool-8-thread-16 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.163:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:45,137 | test | ERROR | pool-8-thread-13 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.150:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:45,217 | test | ERROR | pool-8-thread-18 | [rest_client:_http_request:680] Socket error while connecting to http://172.23.122.162:8091/nodes/self error [Errno 111] Connection refused 2023-04-27 22:30:50,517 | infra | CRITICAL | pool-8-thread-19 | [task:call:6274] mcdMemoryReserved reported in nodes/self is: 9620 2023-04-27 22:30:50,519 | test | CRITICAL | pool-8-thread-19 | [table_view:display:72] Memory quota allocated: +---------------------+---------+ | Service | RAM MiB | +---------------------+---------+ | eventingMemoryQuota | 256 | | cbasMemoryQuota | 1024 | | indexMemoryQuota | 256 | | ftsMemoryQuota | 256 | | memoryQuota | 9520 | +---------------------+---------+ 2023-04-27 22:30:50,628 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:692] RAM quota was defined less than 100 MB: 2023-04-27 22:31:01,430 | test | INFO | MainThread | [basetestcase:initialize_cluster:489] Cluster C1 initialized 2023-04-27 22:31:01,456 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= BaseTestCase setup finished for test #3 test_start_stop_rebalance_with_mutations ========= 2023-04-27 22:31:11,782 | test | INFO | pool-8-thread-20 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+----------------+--------------+-----------------------+ | 172.23.122.150 | kv | 7.2.0-5318-enterprise | 0.997057346763 | Cluster node | active / none | | 172.23.122.139 | None | | | <--- IN --- | | | 172.23.122.141 | None | | | <--- IN --- | | | 172.23.122.163 | None | | | <--- IN --- | | +----------------+----------+-----------------------+----------------+--------------+-----------------------+ 2023-04-27 22:31:11,803 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-04-27 22:31:26,849 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: none, progress: 100 2023-04-27 22:31:26,865 | test | INFO | pool-8-thread-20 | [task:check:480] Rebalance completed with progress: 100% in 15.0829999447 sec 2023-04-27 22:31:27,000 | test | INFO | MainThread | [rebalance_base:collection_setup:222] Creating buckets from spec 2023-04-27 22:31:27,002 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1957] Creating required buckets from template 2023-04-27 22:31:27,341 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2697] Waiting for all collections to be created 2023-04-27 22:31:27,611 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | bucket1 | couchbase | couchstore | 1 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 1 | none | 0 | 0 | 3.91 GiB | 0.0 Byte | 0.0 Byte | 100 | | default | couchbase | magma | 1 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ 2023-04-27 22:31:27,611 | test | INFO | MainThread | [rebalance_base:collection_setup:262] Creating required SDK clients for client_pool 2023-04-27 22:31:38,342 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:31:38,351 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:36:24,059 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 31320 to 35235 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-80_collection-1317_create_ttl=0_create__31320_35235_237148 2023-04-27 22:36:24,075 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 2 docs from 39150 to 43065 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-80_collection-1317_create_ttl=0_create__39150_43065_237152 2023-04-27 22:36:24,075 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 50895 to 54810 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-80_collection-1317_create_ttl=0_create__50895_54810_237158 2023-04-27 22:36:24,345 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 74385 to 78300 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-80_collection-1317_create_ttl=0_create__74385_78300_237170 2023-04-27 22:36:24,805 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 3 docs from 11745 to 15660 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__11745_15660_237218 2023-04-27 22:36:24,815 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 19575 to 23490 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__19575_23490_237222 2023-04-27 22:36:24,815 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 3 docs from 23490 to 27405 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__23490_27405_237224 2023-04-27 22:36:24,815 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 3 docs from 27405 to 31320 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__27405_31320_237226 2023-04-27 22:36:24,816 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 31320 to 35235 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__31320_35235_237228 2023-04-27 22:36:24,816 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 2 docs from 0 to 3915 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-79_collection-1312_create_ttl=0_create__0_3915_237231 2023-04-27 22:36:24,816 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 35235 to 39150 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__35235_39150_237232 2023-04-27 22:36:24,816 | test | ERROR | pool-8-thread-24 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 39150 to 43065 of thread_name LoadDocs_MutateDocsFromSpecTask_223690_default_scope-73_collection-1164_create_ttl=0_create__39150_43065_237236 2023-04-27 22:36:34,275 | test | INFO | pool-8-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****................*........*........******X | 200.000 | | 113.200 | .................X | 113.200 | | 75.400 | ...........X | 75.400 | | 50.200 | .......******..........................*..............................................*....................*****X | 607.400 | | 471.000 | ............................................................................X | 471.000 | | 347.000 | ........................................................X | 347.000 | | 221.000 | ...................................X | 221.000 | | 105.000 | ................******....................*...........*.......*****X | 337.400 | | 263.000 | ..........................................X | 263.000 | | 225.200 | ....................................X | 225.200 | | 200.000 | ...............................******....*..*....*****X | 256.900 | | 211.200 | .................................X | 211.200 | | 186.000 | .............................X | 186.000 | | 135.600 | .....................******..............*............................*...................*****X | 507.900 | | 458.600 | ..........................................................................X | 458.600 | | 333.200 | .....................................................X | 333.200 | | 245.000 | .......................................*****X | 245.000 | | 234.900 | .....................................**..*****X | 248.600 | | 245.400 | .......................................X | 245.400 | | 233.000 | .....................................X | 233.000 | | 231.000 | .....................................*****X | 231.000 | | 199.300 | ...............................X | 199.300 | | 174.500 | ...........................X | 174.500 | | 124.500 | ...................X | 124.500 | | 75.000 | ...........******....*....*....*****X | 150.000 | | 137.000 | .....................X | 137.000 | | 132.200 | ....................*******...X | 145.800 | | 143.000 | ......................X | 143.000 | | 130.400 | ....................*******.....*....******X | 187.400 | | 127.800 | ....................X | 127.800 | | 102.600 | ...............X | 102.600 | | 90.000 | .............******......*......*.....*......*****X | 225.300 | | 165.700 | ..........................X | 165.700 | | 129.100 | ....................X | 129.100 | | 84.700 | ............*****X | 84.700 | | 71.000 | ..........X | 71.000 | | 58.600 | ........X | 58.600 | | 51.700 | .......X | 51.700 | | 48.700 | .......*************X | 51.300 | | 37.600 | .....X | 37.600 | | 25.200 | ...X | 25.200 | | 12.600 | .X | 12.600 | | 0.000 | *******.*..******........*........*......*...*****X | 176.100 | | 128.800 | ....................X | 128.800 | | 123.300 | ...................*.******.*X | 139.500 | | 125.600 | ...................X | 125.600 | | 107.900 | ................*****X | 107.900 | | 100.500 | ...............X | 100.500 | | 77.600 | ...........X | 77.600 | | 72.000 | ..........******...*...*......*....X | 164.900 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2023-04-27 22:36:34,444 | test | INFO | pool-8-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +------------+----------------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+----------------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | *****............................*.................*....................*******.............................*....*..*****X | 118667.800 | | 116085.000 | ................................................................................................X | 116085.000 | | 111238.900 | ............................................................................................X | 111238.900 | | 107883.000 | .........................................................................................X | 107883.000 | | 103480.000 | ......................................................................................*****X | 103480.000 | | 100768.500 | ...................................................................................*..X | 102248.300 | | 100457.400 | ...................................................................................******X | 100457.400 | | 99843.600 | ...................................................................................*...........*....*****X | 118509.100 | | 102715.300 | .....................................................................................X | 102715.300 | | 96501.900 | ................................................................................X | 96501.900 | | 94609.100 | ..............................................................................******......*......X | 108235.600 | | 105378.300 | .......................................................................................*****X | 105378.300 | | 103224.900 | .....................................................................................*..*.*****X | 106794.400 | | 104984.200 | .......................................................................................X | 104984.200 | | 99398.400 | ..................................................................................X | 99398.400 | | 99183.100 | ..................................................................................******.......X | 107581.200 | | 101368.500 | ....................................................................................*...*****X | 104661.700 | | 102856.000 | .....................................................................................X | 102856.000 | | 100835.000 | ...................................................................................*..******X | 103062.800 | | 99259.600 | ..................................................................................*.......X | 107475.100 | | 103187.500 | .....................................................................................*****X | 103187.500 | | 99608.800 | ..................................................................................X | 99608.800 | | 96951.400 | ................................................................................*..*****X | 99258.900 | | 98565.900 | ..................................................................................*X | 98650.700 | | 96028.800 | ...............................................................................******..*.X | 99564.000 | | 96633.400 | ................................................................................******X | 96633.400 | | 93409.100 | .............................................................................*.*.*****X | 95372.700 | | 92440.300 | ............................................................................X | 92440.300 | | 82879.200 | ....................................................................X | 82879.200 | | 76906.400 | ...............................................................*****X | 76906.400 | | 64921.200 | .....................................................X | 64921.200 | | 51268.700 | ..........................................X | 51268.700 | | 47178.900 | ......................................*****X | 47178.900 | | 44127.600 | ....................................X | 44127.600 | | 41064.300 | .................................X | 41064.300 | | 35179.600 | ............................******X | 35179.600 | | 32108.700 | ..........................X | 32108.700 | | 30626.200 | ........................X | 30626.200 | | 29260.700 | .......................********..*******.*******X | 32532.300 | | 31471.200 | .........................X | 31471.200 | | 30404.900 | ........................X | 30404.900 | | 28105.700 | ......................******X | 28105.700 | | 27770.000 | ......................X | 27770.000 | | 27701.200 | ......................X | 27701.200 | | 27618.000 | ......................*****X | 27618.000 | | 27432.800 | ......................X | 27432.800 | | 27289.800 | .....................*.*******.**.*****X | 29668.800 | | 29621.600 | .......................**.X | 29695.200 | | 29520.100 | .......................*******.*******.........*..........*..........................*X | 83422.500 | +------------+----------------------------------------------------------------------------------------------------------------------------+------------+ 2023-04-27 22:36:34,865 | test | INFO | pool-8-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *****..............*........*.......*******...*..*..******...*....*..*****X | 611.200 | | 500.400 | ....................................X | 500.400 | | 400.800 | ............................X | 400.800 | | 300.400 | .....................X | 300.400 | | 200.000 | .............*****X | 200.000 | | 149.100 | ..........X | 149.100 | | 124.000 | ........X | 124.000 | | 98.900 | ......******.......*....*....******.....*.............*........*****X | 649.000 | | 624.100 | .............................................*X | 624.100 | | 526.100 | .....................................*****X | 526.100 | | 514.900 | .....................................X | 514.900 | | 465.100 | .................................X | 465.100 | | 438.800 | ...............................X | 438.800 | | 413.700 | .............................*****X | 413.700 | | 261.800 | ..................X | 261.800 | | 186.500 | ............X | 186.500 | | 126.400 | ........******..........*......................*.........******...........*...........*............................******X | 1350.200 | | 913.100 | ..................................................................X | 913.100 | | 660.900 | ...............................................X | 660.900 | | 435.000 | ...............................******..*X | 465.000 | | 414.800 | .............................*****X | 414.800 | | 405.800 | .............................**......******X | 491.000 | | 359.600 | .........................X | 359.600 | | 284.300 | ....................X | 284.300 | | 209.000 | ..............*****X | 209.000 | | 150.200 | ..........X | 150.200 | | 125.100 | ........X | 125.100 | | 100.000 | ......*******..*..*******...........*..*.*****X | 349.800 | | 300.000 | .....................X | 300.000 | | 225.100 | ...............X | 225.100 | | 200.000 | .............*******.*.*..*****X | 245.000 | | 212.700 | ..............X | 212.700 | | 175.600 | ............X | 175.600 | | 122.900 | ........X | 122.900 | | 82.700 | .....******..*..X | 145.900 | | 139.700 | .........******.*.*.*..******.X | 225.100 | | 218.700 | ...............X | 218.700 | | 217.500 | ...............*****X | 217.500 | | 182.600 | ............X | 182.600 | | 150.200 | ..........********X | 150.200 | | 100.000 | ......X | 100.000 | | 49.800 | ..*****X | 49.800 | | 24.900 | X | 24.900 | | 0.000 | **.******.........................*.............*.........*X | 652.400 | +---------+----------------------------------------------------------------------------------------------------------------------------+----------+ 2023-04-27 22:36:35,045 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ | 172.23.122.141 | kv | 93.9293221427 | 11.74 GiB | 10.02 GiB | 0.0 Byte / 4.10 GiB | 4771073 / 4766023 | 7.2.0-5318-enterprise | | 172.23.122.150 | kv | 92.9622982774 | 11.74 GiB | 9.81 GiB | 0.0 Byte / 0.0 Byte | 4761757 / 4759166 | 7.2.0-5318-enterprise | | 172.23.122.139 | kv | 93.2455086946 | 11.74 GiB | 9.97 GiB | 0.0 Byte / 4.10 GiB | 4814391 / 4800452 | 7.2.0-5318-enterprise | | 172.23.122.163 | kv | 87.1458356001 | 11.74 GiB | 9.89 GiB | 0.0 Byte / 0.0 Byte | 4784342 / 4780133 | 7.2.0-5318-enterprise | +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ 2023-04-27 22:36:36,894 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4910] Validating collection stats and item counts 2023-04-27 22:36:44,006 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4773] Bucket UID mismatch. Expected: 0, Actual: 2 2023-04-27 22:36:44,006 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4927] Bucket manifest UID mismatch! 2023-04-27 22:36:48,046 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4773] Bucket UID mismatch. Expected: 0, Actual: 1 2023-04-27 22:36:48,048 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4927] Bucket manifest UID mismatch! 2023-04-27 22:36:52,042 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4773] Bucket UID mismatch. Expected: 0, Actual: 2 2023-04-27 22:36:52,042 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4927] Bucket manifest UID mismatch! 2023-04-27 22:37:01,401 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ | 172.23.122.141 | kv | 8.14710609887 | 11.74 GiB | 10.06 GiB | 0.0 Byte / 4.10 GiB | 4930800 / 4932600 | 7.2.0-5318-enterprise | | 172.23.122.150 | kv | 8.33134978994 | 11.74 GiB | 9.82 GiB | 0.0 Byte / 0.0 Byte | 4930800 / 4934000 | 7.2.0-5318-enterprise | | 172.23.122.139 | kv | 7.76516863607 | 11.74 GiB | 10.00 GiB | 0.0 Byte / 4.10 GiB | 4931700 / 4924100 | 7.2.0-5318-enterprise | | 172.23.122.163 | kv | 7.32479479762 | 11.74 GiB | 9.89 GiB | 0.0 Byte / 0.0 Byte | 4931700 / 4934300 | 7.2.0-5318-enterprise | +----------------+----------+-----------------+-----------+-----------+---------------------+-------------------+-----------------------+ 2023-04-27 22:37:01,908 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 1 | none | 0 | 100000 | 7.81 GiB | 209.11 MiB | 205.54 MiB | 100 | | bucket2 | couchbase | magma | 1 | none | 0 | 50000 | 3.91 GiB | 292.70 MiB | 232.96 MiB | 100 | | default | couchbase | magma | 1 | none | 0 | 19575000 | 2.00 GiB | 1.35 GiB | 15.65 GiB | 5.25604597701 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2023-04-27 22:37:01,924 | test | INFO | MainThread | [basetestcase:log_setup_status:637] ========= RebalanceBase setup complete for test #3 test_start_stop_rebalance_with_mutations ========= 2023-04-27 22:37:04,069 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:155] Current nodes : [u'ns_1@172.23.122.141', u'ns_1@172.23.122.150', u'ns_1@172.23.122.139', u'ns_1@172.23.122.163'] 2023-04-27 22:37:04,069 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:156] Adding nodes [ip:172.23.122.161 port:8091 ssh_username:root] to cluster 2023-04-27 22:37:04,069 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:157] Removing nodes [] from cluster 2023-04-27 22:37:04,069 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:37:04,075 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:37:04,105 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, -1) 2023-04-27 22:37:04,105 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:37:04,109 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:37:16,607 | test | INFO | pool-8-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.122.141 | kv | 7.2.0-5318-enterprise | 5.8657974589 | Cluster node | active / none | | 172.23.122.150 | kv | 7.2.0-5318-enterprise | 6.47298116211 | Cluster node | active / none | | 172.23.122.139 | kv | 7.2.0-5318-enterprise | 5.82674172391 | Cluster node | active / none | | 172.23.122.163 | kv | 7.2.0-5318-enterprise | 7.32479479762 | Cluster node | active / none | | 172.23.122.161 | None | | | <--- IN --- | | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-04-27 22:37:16,868 | test | INFO | pool-8-thread-15 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-04-27 22:37:26,926 | test | INFO | pool-8-thread-15 | [task:check:421] Rebalance - status: running, progress: 27.2 2023-04-27 22:37:27,513 | test | INFO | MainThread | [cluster_ready_functions:rebalance_reached:867] Rebalance reached >22.52% in 2.02099990845 seconds 2023-04-27 22:37:27,523 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:37:27,526 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:37:27,526 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:185] Stop the rebalance 2023-04-27 22:37:27,710 | test | INFO | MainThread | [rest_client:stop_rebalance:2153] Rebalance stopped 2023-04-27 22:37:27,752 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1214] Running cbcollect on node ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.163 2023-04-27 22:37:37,783 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1217] ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.163 - cbcollect status: True 2023-04-27 22:37:37,783 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1222] Polling active_tasks to check cbcollect status 2023-04-27 22:37:41,986 | test | INFO | pool-8-thread-15 | [task:check:421] Rebalance - status: none, progress: 100 2023-04-27 22:37:42,009 | test | INFO | pool-8-thread-15 | [task:check:480] Rebalance completed with progress: 100% in 25.4019999504 sec 2023-04-27 22:38:53,338 | test | INFO | pool-8-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+----------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------------------------------+----------+ | 144.200 | ......**...*....*.....******...........*......*.....******......*.....*................*******...............*..........*.......*****X | 1995.600 | | 1303.700 | ................................................................X | 1303.700 | | 850.200 | .........................................X | 850.200 | | 445.400 | .....................******...X | 500.800 | | 463.500 | ......................X | 463.500 | | 428.000 | ....................******.*..*..........******X | 693.700 | | 656.500 | ...............................X | 656.500 | | 644.500 | ...............................X | 644.500 | | 620.700 | ..............................*****X | 620.700 | | 556.300 | ..........................X | 556.300 | | 482.900 | .......................X | 482.900 | | 423.100 | ....................******.**..X | 498.100 | +----------+----------------------------------------------------------------------------------------------------------------------------------------+----------+ 2023-04-27 22:38:53,908 | test | INFO | pool-8-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +------------+-------------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+-------------------------------------------------------------------------------------------------------------------------+------------+ | 9348.700 | .....**.........*.......*.........******...................*..................................*..............*******..X | 150315.800 | | 140876.500 | ............................................................................................X | 140876.500 | | 130006.100 | .....................................................................................******..X | 132376.900 | | 115783.500 | ............................................................................X | 115783.500 | | 110407.000 | ........................................................................*****X | 110407.000 | | 95014.500 | ..............................................................*......*......*......******....*...*..******..X | 139136.600 | | 136493.200 | .........................................................................................X | 136493.200 | | 133674.300 | .......................................................................................******........X | 144623.100 | | 138490.700 | ...........................................................................................X | 138490.700 | | 127584.100 | ...................................................................................******X | 127584.100 | | 119813.300 | ..............................................................................*.*.*****X | 122573.400 | | 116836.000 | ............................................................................*..*..X | 122277.700 | +------------+-------------------------------------------------------------------------------------------------------------------------+------------+ 2023-04-27 22:38:55,089 | test | INFO | pool-8-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **............*.......*..........******............................*...................*...........*..........*****..*.X | 882.900 | | 787.400 | ........................................................................................******X | 793.100 | | 586.000 | .................................................................X | 586.000 | | 517.600 | .........................................................X | 517.600 | | 433.600 | ................................................X | 433.600 | | 383.200 | ..........................................****X | 383.200 | | 274.200 | ..............................X | 274.200 | | 262.800 | ............................*.*...*****.......................*..*X | 517.500 | | 492.300 | ......................................................****X | 492.300 | | 459.600 | ...................................................X | 459.600 | | 415.500 | ..............................................*X | 415.500 | | 409.200 | .............................................*****..*...*....*......*....****X | 577.300 | | 424.700 | ...............................................X | 424.700 | | 384.700 | ..........................................X | 384.700 | | 353.200 | .......................................X | 353.200 | | 346.900 | ......................................*****......................*...X | 566.100 | | 534.600 | ...........................................................X | 534.600 | | 503.100 | .......................................................X | 503.100 | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2023-04-27 22:41:39,713 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.139: Copying cbcollect ZIP file to Client 2023-04-27 22:41:40,417 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.141: Copying cbcollect ZIP file to Client 2023-04-27 22:41:41,331 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.150: Copying cbcollect ZIP file to Client 2023-04-27 22:41:42,280 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.161: Copying cbcollect ZIP file to Client 2023-04-27 22:41:42,714 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.163: Copying cbcollect ZIP file to Client 2023-04-27 22:41:43,648 | test | INFO | MainThread | [rebalance_start_stop:tasks_result:63] Waiting for data load to finish 2023-04-27 22:41:44,743 | test | INFO | pool-8-thread-6 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:41:44,865 | test | INFO | pool-8-thread-9 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:41:49,862 | test | INFO | pool-8-thread-6 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:41:49,983 | test | INFO | pool-8-thread-9 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:41:54,982 | test | INFO | pool-8-thread-6 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:41:55,099 | test | INFO | pool-8-thread-9 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:00,101 | test | INFO | pool-8-thread-6 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:00,217 | test | INFO | pool-8-thread-9 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:05,220 | test | INFO | pool-8-thread-6 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:05,334 | test | INFO | pool-8-thread-9 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:15,349 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:42:15,351 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:42:15,351 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:202] Rebalance is still required. Verifying the data in the buckets 2023-04-27 22:42:16,344 | test | INFO | pool-8-thread-18 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:16,454 | test | INFO | pool-8-thread-5 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:21,461 | test | INFO | pool-8-thread-18 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:21,571 | test | INFO | pool-8-thread-5 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:26,579 | test | INFO | pool-8-thread-18 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:26,686 | test | INFO | pool-8-thread-5 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:31,697 | test | INFO | pool-8-thread-18 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:31,799 | test | INFO | pool-8-thread-5 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:36,816 | test | INFO | pool-8-thread-18 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:36,915 | test | INFO | pool-8-thread-5 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:42:41,917 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:42:41,924 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:42:41,954 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, -1) 2023-04-27 22:42:41,954 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:42:41,960 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:42:47,959 | test | INFO | pool-8-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.122.141 | kv | 7.2.0-5318-enterprise | 3.69343623992 | Cluster node | active / none | | 172.23.122.161 | kv | 7.2.0-5318-enterprise | 1.91913274722 | Cluster node | active / none | | 172.23.122.150 | kv | 7.2.0-5318-enterprise | 4.01819502184 | Cluster node | active / none | | 172.23.122.139 | kv | 7.2.0-5318-enterprise | 3.66370678979 | Cluster node | active / none | | 172.23.122.163 | kv | 7.2.0-5318-enterprise | 3.81906406519 | Cluster node | active / none | | 172.23.122.162 | None | | | <--- IN --- | | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-04-27 22:42:48,086 | test | INFO | pool-8-thread-11 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-04-27 22:42:58,130 | test | INFO | pool-8-thread-11 | [task:check:421] Rebalance - status: running, progress: 27.67 2023-04-27 22:43:08,029 | test | INFO | MainThread | [cluster_ready_functions:rebalance_reached:867] Rebalance reached >46.25% in 4.08800005913 seconds 2023-04-27 22:43:08,039 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:43:08,040 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:43:08,042 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:185] Stop the rebalance 2023-04-27 22:43:08,072 | test | WARNING | MainThread | [rest_client:stop_rebalance:2148] Rebalance not stopped after 1 sec 2023-04-27 22:43:08,229 | test | INFO | pool-8-thread-11 | [task:check:421] Rebalance - status: running, progress: 53.73 2023-04-27 22:43:09,095 | test | INFO | MainThread | [rest_client:stop_rebalance:2153] Rebalance stopped 2023-04-27 22:43:09,160 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1214] Running cbcollect on node ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 2023-04-27 22:43:19,184 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1217] ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 - cbcollect status: True 2023-04-27 22:43:19,184 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1222] Polling active_tasks to check cbcollect status 2023-04-27 22:43:23,316 | test | INFO | pool-8-thread-11 | [task:check:421] Rebalance - status: none, progress: 100 2023-04-27 22:43:23,352 | test | INFO | pool-8-thread-11 | [task:check:480] Rebalance completed with progress: 100% in 35.393999815 sec 2023-04-27 22:45:11,720 | test | ERROR | pool-8-thread-1 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 1957 to 3914 of thread_name LoadDocs_MutateDocsFromSpecTask_260481_default__default_collection-1101_update_ttl=0_update__1957_3914_265275 2023-04-27 22:45:11,721 | test | ERROR | pool-8-thread-1 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1 docs from 7828 to 9785 of thread_name LoadDocs_MutateDocsFromSpecTask_260481_default__default_collection-1101_update_ttl=0_update__7828_9785_265281 2023-04-27 22:45:39,368 | test | INFO | pool-8-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***......*.....*......******X | 2105.700 | | 2074.100 | ................X | 2074.100 | | 2027.300 | ................******..*..........*...................*...******................*.....*..........*......*****X | 10381.900 | | 9158.200 | ............................................................................X | 9158.200 | | 8451.200 | ......................................................................X | 8451.200 | | 7372.300 | .............................................................X | 7372.300 | | 6934.300 | .........................................................****X | 6934.300 | | 6803.300 | ........................................................X | 6803.300 | | 6371.000 | .....................................................X | 6371.000 | | 6235.600 | ...................................................*.*****X | 6272.900 | | 6255.000 | ....................................................X | 6255.000 | | 6097.000 | ..................................................X | 6097.000 | | 5817.100 | ................................................X | 5817.100 | | 5661.400 | ...............................................****X | 5661.400 | | 5596.700 | ..............................................*.*.*.*******X | 5949.500 | | 5934.300 | .................................................****X | 5934.300 | | 5931.500 | .................................................**.*....******...................*.........*............*....****X | 11665.300 | | 11659.200 | ..................................................................................................*X | 11673.000 | | 11671.100 | ..................................................................................................*******.X | 11771.100 | | 11755.200 | ..................................................................................................X | 11755.200 | | 11723.200 | ..................................................................................................X | 11723.200 | | 11717.400 | ..................................................................................................****X | 11717.400 | | 11693.800 | ..................................................................................................*******X | 11715.100 | | 11658.600 | ..................................................................................................X | 11658.600 | | 11427.500 | ................................................................................................X | 11427.500 | | 11380.500 | ...............................................................................................X | 11380.500 | | 11254.800 | ..............................................................................................****X | 11254.800 | | 10573.400 | ........................................................................................X | 10573.400 | | 8382.000 | ......................................................................X | 8382.000 | | 3796.500 | ...............................X | 3796.500 | | 2924.800 | .......................****X | 2924.800 | | 2771.400 | ......................X | 2771.400 | | 2360.400 | ...................*...*.****X | 2915.100 | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ 2023-04-27 22:45:39,792 | test | INFO | pool-8-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +------------+--------------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+--------------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | ***..............*...........*.............******......................................*.............*..........******.X | 147417.600 | | 115708.200 | .............................................................................X | 115708.200 | | 100583.000 | ...................................................................******X | 100583.000 | | 86873.100 | .........................................................X | 86873.100 | | 79304.500 | ....................................................X | 79304.500 | | 73086.700 | ................................................*****X | 73086.700 | | 56880.600 | .....................................*......*....*******X | 72116.500 | | 71732.800 | ...............................................X | 71732.800 | | 71507.300 | ...............................................*****X | 71507.300 | | 71044.100 | ...............................................X | 71044.100 | | 70566.000 | ..............................................X | 70566.000 | | 70042.900 | ..............................................******X | 70042.900 | | 64389.900 | ..........................................X | 64389.900 | | 61265.300 | ........................................X | 61265.300 | | 58590.400 | ......................................*****X | 58590.400 | | 58373.000 | ......................................X | 58373.000 | | 58130.200 | ......................................********X | 58530.300 | | 57797.300 | ......................................X | 57797.300 | | 56396.100 | .....................................*****X | 56396.100 | | 42135.000 | ...........................X | 42135.000 | | 35661.100 | .......................X | 35661.100 | | 29878.500 | ...................*****X | 29878.500 | | 29565.600 | ...................X | 29565.600 | | 29447.900 | ..................*X | 29474.900 | | 29420.700 | ..................******.*X | 29723.200 | | 29717.500 | ...................******X | 29717.500 | | 29322.500 | ..................X | 29322.500 | | 29151.800 | ..................X | 29151.800 | | 29010.600 | ..................*****X | 29010.600 | | 28779.200 | ..................X | 28779.200 | | 28419.000 | ..................X | 28419.000 | | 28369.400 | ..................******.............*.............*..................................*******.............X | 136773.500 | | 134045.400 | .........................................................................................X | 134045.400 | | 127130.300 | .....................................................................................*****X | 127130.300 | +------------+--------------------------------------------------------------------------------------------------------------------------+------------+ 2023-04-27 22:45:40,586 | test | INFO | pool-8-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------------------------+----------+ | 79.800 | ***......*.....*...*....*****...*.**...******........*.......**...........*****.*...............*.*.........*****X | 5636.700 | | 5029.500 | ....................................................................X | 5029.500 | | 4520.500 | .............................................................X | 4520.500 | | 4451.600 | ............................................................X | 4451.600 | | 3478.700 | ...............................................****X | 3478.700 | | 3406.400 | ..............................................X | 3406.400 | | 2936.500 | .......................................X | 2936.500 | | 2905.300 | .......................................X | 2905.300 | | 2856.700 | ......................................*****.*.X | 3015.500 | | 3011.300 | ........................................X | 3011.300 | | 2979.500 | ........................................X | 2979.500 | | 2929.800 | .......................................*****..*...**.....*****.*.*X | 3766.400 | | 3687.700 | ..................................................****X | 3687.700 | | 3664.500 | .................................................*.*.*.......******...............*..........*..*..........*****.*.*.*******.X | 7192.900 | | 7167.900 | ..................................................................................................X | 7167.900 | | 7162.700 | ..................................................................................................X | 7162.700 | | 7136.300 | ..................................................................................................****X | 7136.300 | | 7114.300 | .................................................................................................X | 7114.300 | | 7059.800 | .................................................................................................X | 7059.800 | | 7046.600 | ................................................................................................X | 7046.600 | | 6911.400 | ...............................................................................................****X | 6911.400 | | 6894.100 | ..............................................................................................X | 6894.100 | | 6763.500 | .............................................................................................*X | 6768.900 | | 6653.500 | ...........................................................................................****X | 6653.500 | | 6559.100 | ..........................................................................................X | 6559.100 | | 5432.200 | ..........................................................................X | 5432.200 | | 4159.200 | ........................................................X | 4159.200 | | 2330.800 | ...............................****X | 2330.800 | | 2197.300 | .............................X | 2197.300 | | 1747.500 | .......................X | 1747.500 | | 1592.200 | .....................*..*****.X | 1820.200 | +----------+--------------------------------------------------------------------------------------------------------------------------------+----------+ 2023-04-27 22:47:36,207 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.139: Copying cbcollect ZIP file to Client 2023-04-27 22:47:36,858 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.141: Copying cbcollect ZIP file to Client 2023-04-27 22:47:37,861 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.150: Copying cbcollect ZIP file to Client 2023-04-27 22:47:39,043 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.161: Copying cbcollect ZIP file to Client 2023-04-27 22:47:39,766 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.162: Copying cbcollect ZIP file to Client 2023-04-27 22:47:40,239 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.163: Copying cbcollect ZIP file to Client 2023-04-27 22:47:41,073 | test | INFO | MainThread | [rebalance_start_stop:tasks_result:63] Waiting for data load to finish 2023-04-27 22:47:41,075 | test | CRITICAL | MainThread | [rebalance_start_stop:tasks_result:66] Seeing failures in mutatate_from_spec 2023-04-27 22:47:41,107 | test | ERROR | pool-8-thread-7 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 13 docs from 39150 to 40716 of thread_name LoadDocs_MutateDocsFromSpecTask_260482_default_scope-79_collection-1298_cont_update_ttl=0_update__39150_40716_260485 2023-04-27 22:47:42,332 | test | INFO | pool-8-thread-19 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:42,509 | test | INFO | pool-8-thread-1 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:47,456 | test | INFO | pool-8-thread-19 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:47,625 | test | INFO | pool-8-thread-1 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:52,575 | test | INFO | pool-8-thread-19 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:52,744 | test | INFO | pool-8-thread-1 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:57,696 | test | INFO | pool-8-thread-19 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:47:57,864 | test | INFO | pool-8-thread-1 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:02,815 | test | INFO | pool-8-thread-19 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:02,983 | test | INFO | pool-8-thread-1 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:13,005 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:48:13,006 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:48:13,006 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:202] Rebalance is still required. Verifying the data in the buckets 2023-04-27 22:48:14,168 | test | INFO | pool-8-thread-22 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:14,335 | test | INFO | pool-8-thread-29 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:19,288 | test | INFO | pool-8-thread-22 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:19,453 | test | INFO | pool-8-thread-29 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:24,407 | test | INFO | pool-8-thread-22 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:24,572 | test | INFO | pool-8-thread-29 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:29,526 | test | INFO | pool-8-thread-22 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:29,691 | test | INFO | pool-8-thread-29 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:34,644 | test | INFO | pool-8-thread-22 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:34,809 | test | INFO | pool-8-thread-29 | [common_lib:sleep:23] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 18, in cbstats.main() File "/opt/couchbase/lib/python/cbstats.py", line 1055, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats.py", line 50, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2023-04-27 22:48:39,812 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:48:39,816 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:48:39,842 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, -1) 2023-04-27 22:48:39,844 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:48:39,846 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:48:43,766 | test | INFO | pool-8-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.122.162 | kv | 7.2.0-5318-enterprise | 2.05740369824 | Cluster node | active / none | | 172.23.122.141 | kv | 7.2.0-5318-enterprise | 4.12191476118 | Cluster node | active / none | | 172.23.122.161 | kv | 7.2.0-5318-enterprise | 1.89587819436 | Cluster node | active / none | | 172.23.122.150 | kv | 7.2.0-5318-enterprise | 4.10853515542 | Cluster node | active / none | | 172.23.122.139 | kv | 7.2.0-5318-enterprise | 3.88704439036 | Cluster node | active / none | | 172.23.122.163 | kv | 7.2.0-5318-enterprise | 4.26279897272 | Cluster node | active / none | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-04-27 22:48:44,233 | test | INFO | pool-8-thread-30 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-04-27 22:48:54,473 | test | INFO | pool-8-thread-30 | [task:check:421] Rebalance - status: running, progress: 66.72 2023-04-27 22:49:04,260 | test | INFO | MainThread | [cluster_ready_functions:rebalance_reached:867] Rebalance reached >69.3% in 2.02399992943 seconds 2023-04-27 22:49:04,272 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:49:04,275 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:49:04,275 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:185] Stop the rebalance 2023-04-27 22:49:04,309 | test | WARNING | MainThread | [rest_client:stop_rebalance:2148] Rebalance not stopped after 1 sec 2023-04-27 22:49:05,332 | test | INFO | MainThread | [rest_client:stop_rebalance:2153] Rebalance stopped 2023-04-27 22:49:05,407 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1214] Running cbcollect on node ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 2023-04-27 22:49:09,546 | test | INFO | pool-8-thread-30 | [task:check:421] Rebalance - status: none, progress: 100 2023-04-27 22:49:09,572 | test | INFO | pool-8-thread-30 | [task:check:480] Rebalance completed with progress: 100% in 25.8059999943 sec 2023-04-27 22:49:15,430 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1217] ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 - cbcollect status: True 2023-04-27 22:49:15,430 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1222] Polling active_tasks to check cbcollect status 2023-04-27 22:50:31,811 | test | INFO | pool-8-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.......*.........................*...........*............*****.....*.............*X | 254.200 | | 246.400 | ......................................................................*****X | 246.400 | | 220.800 | ..............................................................**...*...*****.......*...X | 275.000 | | 267.400 | ............................................................................X | 267.400 | | 263.200 | ..........................................................................*****....*....*.....*****X | 307.500 | | 286.300 | .................................................................................X | 286.300 | | 216.700 | .............................................................X | 216.700 | | 184.500 | ....................................................X | 184.500 | | 176.100 | .................................................*****..........*............*....................*......****X | 341.900 | | 338.200 | ................................................................................................*...X | 346.800 | | 342.400 | .................................................................................................X | 342.400 | | 336.300 | ...............................................................................................*****X | 336.300 | | 243.000 | .....................................................................X | 243.000 | | 216.600 | .............................................................X | 216.600 | | 193.000 | ......................................................X | 193.000 | | 175.400 | .................................................*****.X | 179.100 | | 173.100 | ................................................*..X | 177.200 | | 162.000 | .............................................****X | 162.000 | | 147.200 | .........................................X | 147.200 | | 117.900 | ................................X | 117.900 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2023-04-27 22:50:32,517 | test | INFO | pool-8-thread-25 | [table_view:display:72] Ops trend for bucket 'default' +------------+-----------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+-----------------------------------------------------------------------------------------------------------------------+------------+ | 3229.800 | .****................*..................*......................................*******...............*..*.....*.****X | 148590.900 | | 137917.600 | .........................................................................................X | 137917.600 | | 130812.300 | .....................................................................................X | 130812.300 | | 127602.100 | ..................................................................................X | 127602.100 | | 125848.000 | .................................................................................*..*****.........**.......X | 152098.700 | | 151021.600 | ..................................................................................................****X | 151021.600 | | 140415.200 | ...........................................................................................X | 140415.200 | | 128338.200 | ...................................................................................X | 128338.200 | | 123589.000 | ................................................................................X | 123589.000 | | 123256.200 | ................................................................................******.X | 125352.800 | | 124818.700 | .................................................................................*..*****X | 128305.500 | | 128239.800 | ...................................................................................X | 128239.800 | | 125451.000 | .................................................................................X | 125451.000 | | 120755.900 | ..............................................................................X | 120755.900 | | 120278.800 | ..............................................................................****X | 120278.800 | | 119857.300 | .............................................................................*.*..*.****X | 124848.000 | | 124257.500 | ................................................................................X | 124257.500 | | 122437.600 | ...............................................................................X | 122437.600 | | 120946.200 | ..............................................................................X | 120946.200 | | 120073.800 | .............................................................................X | 120073.800 | | 119898.200 | .............................................................................*****........*......X | 140135.200 | | 137502.700 | .........................................................................................*.****X | 138453.000 | | 119475.400 | .............................................................................X | 119475.400 | | 105687.500 | ....................................................................X | 105687.500 | +------------+-----------------------------------------------------------------------------------------------------------------------+------------+ 2023-04-27 22:50:33,686 | test | INFO | pool-8-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+----------+ | 145.700 | ..****....*......................*....*.........*****X | 1795.400 | | 1680.700 | ......................................X | 1680.700 | | 1547.400 | ...................................X | 1547.400 | | 1519.900 | ..................................******.......*....X | 1949.000 | | 1937.100 | ............................................*...****X | 2041.900 | | 1988.500 | .............................................*..*..*......*****............*.....*..*.........****X | 3575.800 | | 3520.900 | ..................................................................................X | 3520.900 | | 3291.900 | ............................................................................X | 3291.900 | | 3182.400 | ..........................................................................X | 3182.400 | | 2820.800 | .................................................................****X | 2820.800 | | 2815.700 | .................................................................X | 2815.700 | | 2689.900 | ..............................................................X | 2689.900 | | 2651.700 | .............................................................*.*****X | 2674.600 | | 2672.300 | ..............................................................X | 2672.300 | | 2600.100 | ............................................................X | 2600.100 | | 2476.800 | .........................................................****X | 2476.800 | | 2461.400 | .........................................................**..**....****X | 2729.300 | | 2339.400 | ......................................................X | 2339.400 | | 2030.500 | ..............................................X | 2030.500 | | 1966.100 | .............................................X | 1966.100 | | 1567.400 | ....................................*****...............................*...........................*.....X | 4232.800 | +----------+-------------------------------------------------------------------------------------------------------------+----------+ 2023-04-27 22:53:17,650 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.139: Copying cbcollect ZIP file to Client 2023-04-27 22:53:18,276 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.141: Copying cbcollect ZIP file to Client 2023-04-27 22:53:19,049 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.150: Copying cbcollect ZIP file to Client 2023-04-27 22:53:19,881 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.161: Copying cbcollect ZIP file to Client 2023-04-27 22:53:20,476 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.162: Copying cbcollect ZIP file to Client 2023-04-27 22:53:20,969 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.163: Copying cbcollect ZIP file to Client 2023-04-27 22:53:21,890 | test | INFO | MainThread | [rebalance_start_stop:tasks_result:63] Waiting for data load to finish 2023-04-27 22:53:28,894 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:53:28,895 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:53:28,895 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:202] Rebalance is still required. Verifying the data in the buckets 2023-04-27 22:53:30,559 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:53:30,565 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:53:30,592 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, -1) 2023-04-27 22:53:30,592 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:53:30,596 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:53:41,253 | test | INFO | pool-8-thread-20 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.122.162 | kv | 7.2.0-5318-enterprise | 15.8058939206 | Cluster node | active / none | | 172.23.122.141 | kv | 7.2.0-5318-enterprise | 89.3640820675 | Cluster node | active / none | | 172.23.122.161 | kv | 7.2.0-5318-enterprise | 21.5537644962 | Cluster node | active / none | | 172.23.122.150 | kv | 7.2.0-5318-enterprise | 91.8201574006 | Cluster node | active / none | | 172.23.122.139 | kv | 7.2.0-5318-enterprise | 88.949182298 | Cluster node | active / none | | 172.23.122.163 | kv | 7.2.0-5318-enterprise | 73.4850976507 | Cluster node | active / none | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-04-27 22:53:42,102 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-04-27 22:53:52,522 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: running, progress: 69.37 2023-04-27 22:54:02,575 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: running, progress: 73.76 2023-04-27 22:54:12,663 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: running, progress: 77.63 2023-04-27 22:54:22,707 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: running, progress: 81.74 2023-04-27 22:54:22,762 | test | INFO | MainThread | [cluster_ready_functions:rebalance_reached:867] Rebalance reached >81.03% in 22.3370001316 seconds 2023-04-27 22:54:22,775 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:54:22,776 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:54:22,776 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:185] Stop the rebalance 2023-04-27 22:54:22,801 | test | WARNING | MainThread | [rest_client:stop_rebalance:2148] Rebalance not stopped after 1 sec 2023-04-27 22:54:23,822 | test | INFO | MainThread | [rest_client:stop_rebalance:2153] Rebalance stopped 2023-04-27 22:54:23,864 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1214] Running cbcollect on node ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 2023-04-27 22:54:33,891 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1217] ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 - cbcollect status: True 2023-04-27 22:54:33,891 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1222] Polling active_tasks to check cbcollect status 2023-04-27 22:54:37,785 | test | INFO | pool-8-thread-20 | [task:check:421] Rebalance - status: none, progress: 100 2023-04-27 22:54:37,819 | test | INFO | pool-8-thread-20 | [task:check:480] Rebalance completed with progress: 100% in 56.5650000572 sec 2023-04-27 22:55:12,630 | test | INFO | pool-8-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****.....*..............*......*...................******...............................*............*........*.....****X | 450.000 | | 398.200 | .......................................................................................X | 398.200 | | 204.600 | ............................................X | 204.600 | | 147.200 | ...............................X | 147.200 | | 105.700 | ......................*****.......*................*...............*......******........*..*......*.....****X | 399.700 | | 346.800 | ............................................................................X | 346.800 | | 320.400 | ......................................................................X | 320.400 | | 271.200 | ...........................................................X | 271.200 | | 212.600 | ..............................................****X | 212.600 | | 190.400 | .........................................X | 190.400 | | 121.600 | ..........................*.*......*****.............*................................*.........*....*.....****X | 437.000 | | 294.300 | ................................................................X | 294.300 | | 264.700 | .........................................................X | 264.700 | +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ 2023-04-27 22:55:13,125 | test | INFO | pool-8-thread-25 | [table_view:display:72] Ops trend for bucket 'default' +------------+--------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+--------------------------------------------------------------------------------------------------------------------+------------+ | 9144.400 | ....*****............*............................*...............*...*****.....*......*...*.*****X | 143413.400 | | 142190.100 | .............................................................................X | 142190.100 | | 130624.600 | ......................................................................X | 130624.600 | | 122594.500 | ..................................................................*.****X | 123989.600 | | 115212.700 | ..............................................................X | 115212.700 | | 110646.000 | ...........................................................X | 110646.000 | | 102713.000 | .......................................................X | 102713.000 | | 99900.800 | .....................................................*****...*.......*........*.......*****.....X | 153238.000 | | 149092.600 | .................................................................................X | 149092.600 | | 142196.500 | .............................................................................X | 142196.500 | | 134458.200 | .........................................................................****X | 134458.200 | | 124938.800 | ...................................................................X | 124938.800 | | 102309.900 | .......................................................X | 102309.900 | | 99910.500 | ......................................................******.....*.................*...............*....*****....X | 181647.900 | | 158894.500 | ......................................................................................X | 158894.500 | | 144002.700 | ..............................................................................X | 144002.700 | +------------+--------------------------------------------------------------------------------------------------------------------+------------+ 2023-04-27 22:55:13,233 | test | INFO | pool-8-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------------+----------+ | 705.200 | ......*****...*.............*..*.....*****.X | 2796.700 | | 2790.400 | ..............................*X | 2798.900 | | 2721.600 | .............................******...*.....X | 3361.600 | | 3293.600 | ....................................*...*****.....*....*.*...........****X | 5407.600 | | 4900.800 | ......................................................X | 4900.800 | | 4205.600 | ..............................................X | 4205.600 | | 4091.500 | .............................................X | 4091.500 | | 2628.200 | ............................****X | 2628.200 | | 2530.100 | ...........................X | 2530.100 | | 2357.500 | .........................*..*.........******.........................*................*...*...................****X | 8822.600 | | 6411.400 | .......................................................................X | 6411.400 | | 4765.100 | .....................................................X | 4765.100 | | 4418.000 | .................................................X | 4418.000 | | 2327.700 | .........................*****.*..........*............X | 4406.800 | +----------+---------------------------------------------------------------------------------------------------------------------+----------+ 2023-04-27 22:58:25,821 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.139: Copying cbcollect ZIP file to Client 2023-04-27 22:58:26,470 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.141: Copying cbcollect ZIP file to Client 2023-04-27 22:58:27,203 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.150: Copying cbcollect ZIP file to Client 2023-04-27 22:58:28,048 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.161: Copying cbcollect ZIP file to Client 2023-04-27 22:58:28,638 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.162: Copying cbcollect ZIP file to Client 2023-04-27 22:58:29,216 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.163: Copying cbcollect ZIP file to Client 2023-04-27 22:58:30,009 | test | INFO | MainThread | [rebalance_start_stop:tasks_result:63] Waiting for data load to finish 2023-04-27 22:58:36,921 | test | ERROR | MainThread | [rest_client:_http_request:676] POST http://172.23.122.150:8091/diag/eval/ body: ns_orchestrator:needs_rebalance() headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2023-04-27 22:58:36,923 | test | CRITICAL | MainThread | [cluster_ready_functions:is_cluster_rebalanced:877] Can't define if cluster balanced 2023-04-27 22:58:36,924 | test | INFO | MainThread | [rebalance_start_stop:test_start_stop_rebalance_with_mutations:202] Rebalance is still required. Verifying the data in the buckets 2023-04-27 22:58:38,614 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:58:38,619 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:58:38,645 | test | INFO | MainThread | [collections_base:start_history_retention_data_load:198] Starting dedupe updates load (2, -1) 2023-04-27 22:58:38,647 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5167] Performing scope/collection specific operations 2023-04-27 22:58:38,651 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5257] Done Performing scope/collection specific operations 2023-04-27 22:58:42,198 | test | INFO | pool-8-thread-12 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.122.162 | kv | 7.2.0-5318-enterprise | 73.5012522222 | Cluster node | active / none | | 172.23.122.141 | kv | 7.2.0-5318-enterprise | 89.148121171 | Cluster node | active / none | | 172.23.122.161 | kv | 7.2.0-5318-enterprise | 78.9377241029 | Cluster node | active / none | | 172.23.122.150 | kv | 7.2.0-5318-enterprise | 90.276441439 | Cluster node | active / none | | 172.23.122.139 | kv | 7.2.0-5318-enterprise | 88.4514005236 | Cluster node | active / none | | 172.23.122.163 | kv | 7.2.0-5318-enterprise | 55.6978492244 | Cluster node | active / none | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-04-27 22:58:42,660 | test | INFO | pool-8-thread-12 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-04-27 22:58:52,845 | test | INFO | pool-8-thread-12 | [task:check:421] Rebalance - status: running, progress: 69.93 2023-04-27 22:59:02,882 | test | INFO | pool-8-thread-12 | [task:check:421] Rebalance - status: running, progress: 73.49 2023-04-27 22:59:12,921 | test | INFO | pool-8-thread-12 | [task:check:421] Rebalance - status: running, progress: 78.14 2023-04-27 22:59:19,701 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 10 docs from 0 to 1957 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__0_1957_324352 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 15 docs from 1957 to 3914 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__1957_3914_324354 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 18 docs from 3914 to 5871 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__3914_5871_324356 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 13 docs from 5871 to 7828 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__5871_7828_324358 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 12 docs from 7828 to 9785 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__7828_9785_324360 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 33 docs from 9785 to 11742 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__9785_11742_324362 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 17 docs from 11742 to 13699 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__11742_13699_324364 2023-04-27 22:59:19,703 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 15 docs from 13699 to 15656 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__13699_15656_324366 2023-04-27 22:59:19,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 34 docs from 15656 to 17613 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__15656_17613_324368 2023-04-27 22:59:19,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 30 docs from 17613 to 19570 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__17613_19570_324370 2023-04-27 22:59:19,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 28 docs from 19570 to 21527 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__19570_21527_324372 2023-04-27 22:59:19,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 20 docs from 21527 to 23484 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__21527_23484_324374 2023-04-27 22:59:19,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 25 docs from 23484 to 25441 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__23484_25441_324376 2023-04-27 22:59:19,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 28 docs from 25441 to 27398 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__25441_27398_324378 2023-04-27 22:59:19,714 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 26 docs from 27398 to 29355 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__27398_29355_324380 2023-04-27 22:59:19,714 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 18 docs from 29355 to 31312 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__29355_31312_324382 2023-04-27 22:59:19,714 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 17 docs from 31312 to 33269 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__31312_33269_324384 2023-04-27 22:59:19,714 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 18 docs from 33269 to 35226 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__33269_35226_324386 2023-04-27 22:59:19,714 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 14 docs from 35226 to 37183 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__35226_37183_324388 Some events could not be published because the queue was (likely temporarily) over capacity: {"RequestRetryScheduledEvent":{"sampleEvent":"RequestRetryScheduledEvent{severity=DEBUG, category=com.couchbase.request, duration=PT0.008S, created=2023-04-28T05:59:19.798Z, description=Request UpsertRequest retry scheduled per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context={completed=false, coreId=0xaef52ee000000056, idempotent=false, lastDispatchedTo=172.23.122.141, requestId=336844193, requestType=UpsertRequest, retried=5, retryReasons=[ENDPOINT_NOT_AVAILABLE], service={bucket=default, collection=collection-1197, documentId=test_collections-6485, opaque=0x14154fab, scope=scope-75, type=kv, vbucket=291}, timeoutMs=120000, timings={encodingMicros=2}}, cause=null}","totalDropCount":880},"EndpointConnectionFailedEvent":{"sampleEvent":"EndpointConnectionFailedEvent{severity=WARN, category=com.couchbase.endpoint, duration=PT0.010728515S, created=2023-04-28T05:59:19.798Z, description=Connect attempt 3 failed because of : finishConnect(..) failed: Connection refused: /172.23.122.141:11210 - Check server ports and cluster encryption setting., context={bucket=bucket1, channelId=AEF52EE000000047/000000008793E6B6, circuitBreaker=DISABLED, coreId=0xaef52ee000000047, local=172.23.123.170:49970, remote=172.23.122.141:11210, type=KV}, cause=com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.122.141:11210 - Check server ports and cluster encryption setting.}","totalDropCount":69},"EndpointStateChangedEvent":{"sampleEvent":"EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, created=2023-04-28T05:59:19.798Z, description=Endpoint changed state from DISCONNECTED to CONNECTING, context={bucket=bucket2, channelId=AEF52EE000000041/00000000BCC4E15A, circuitBreaker=DISABLED, coreId=0xaef52ee000000041, local=172.23.123.170:48050, remote=172.23.122.141:11210, type=KV}, cause=null}","totalDropCount":122},"ServiceStateChangedEvent":{"sampleEvent":"ServiceStateChangedEvent{severity=DEBUG, category=com.couchbase.service, duration=PT0S, created=2023-04-28T05:59:19.798Z, description=Service changed state from CONNECTING to DISCONNECTED, context={coreId=0xaef52ee000000043, remote=172.23.122.141:11210, type=KV}, cause=null}","totalDropCount":6},"SaslMechanismsSelectedEvent":{"sampleEvent":"SaslMechanismsSelectedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0S, created=2023-04-28T05:59:19.798Z, description=SASL Mechanism SCRAM_SHA512 selected. Allowed: [SCRAM_SHA1, SCRAM_SHA256, SCRAM_SHA512], context={bucket=bucket1, coreId=0xaef52ee00000004d, local=/172.23.123.170:47432, remote=172.23.122.141/172.23.122.141:11210}, cause=null}","totalDropCount":2}} 2023-04-27 22:59:21,565 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1556] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0922e99a89c59a425f3a98944f44d1ac', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=3ea5652b0c733b191fbc2da02a7777c4', u'status': u'notRunning'} - rebalance failed 2023-04-27 22:59:21,605 | test | INFO | MainThread | [rest_client:print_UI_logs:2738] Latest logs from UI on 172.23.122.150: 2023-04-27 22:59:21,607 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661561238L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.238Z', u'text': u'Bucket "bucket1" loaded on node \'ns_1@172.23.122.141\' in 0 seconds.'} 2023-04-27 22:59:21,607 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661559689L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.689Z', u'text': u"Service 'memcached' exited with status 139. Restarting. Messages:\n #9 /opt/couchbase/bin/memcached() [0x400000+0x23b83b]\n #10 /opt/couchbase/bin/memcached() [0x400000+0x20e0cb]\n #11 /opt/couchbase/bin/memcached() [0x400000+0x20e160]\n #12 /opt/couchbase/bin/memcached() [0x400000+0x1ef543]\n #13 /opt/couchbase/bin/memcached() [0x400000+0x1ef7f2]\n #14 /opt/couchbase/bin/memcached() [0x400000+0x1efe7b]\n #15 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f9a4e093000+0xed7e]\n #16 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f9a4e093000+0x17cf1]\n #17 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7f9a4e093000+0x1853f]\n #18 /opt/couchbase/bin/memcached() [0x400000+0x824927]\n #19 /opt/couchbase/bin/memcached() [0x400000+0x824df6]\n #20 /opt/couchbase/bin/memcached() [0x400000+0x8267b6]\n #21 /opt/couchbase/bin/memcached() [0x400000+0x1bbfc9]\n #22 /opt/couchbase/bin/memcached() [0x400000+0x7a9409]\n #23 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f9a4ebd3000+0x7fa3]\n #24 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f9a4c9aa000+0xf8eff]"} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661559644L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.644Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.23052.15>,\n {{{{{child_interrupted,\n {\'EXIT\',<26289.7918.4>,socket_closed}},\n [{dcp_replicator,spawn_and_wait,1,\n [{file,"src/dcp_replicator.erl"},\n {line,358}]},\n {dcp_replicator,handle_call,3,\n [{file,"src/dcp_replicator.erl"},\n {line,152}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,721}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,750}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [<26289.7916.4>,get_partitions,infinity]}},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,63},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.122.162\'},\n {if_rebalance,<0.10041.15>,\n {dcp_takeover,\'ns_1@172.23.122.150\',514}},\n infinity]}}}}}.\nRebalance Operation Id = e98e17eddc3d7b874e5da9ee8fbccfa8'} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661559617L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.617Z', u'text': u"Control connection to memcached on 'ns_1@172.23.122.141' disconnected. Check logs for details."} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661559610L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.610Z', u'text': u'Worker <0.22554.15> (for action {move,{514,\n [\'ns_1@172.23.122.150\',\n \'ns_1@172.23.122.139\'],\n [\'ns_1@172.23.122.162\',\n \'ns_1@172.23.122.161\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.23052.15>,\n {{{{{child_interrupted,\n {\'EXIT\',\n <26289.7918.4>,\n socket_closed}},\n [{dcp_replicator,\n spawn_and_wait,\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 358}]},\n {dcp_replicator,\n handle_call,\n 3,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 152}]},\n {gen_server,\n try_handle_call,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 721}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 750}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [<26289.7916.4>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,\n 63},\n infinity]}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.122.162\'},\n {if_rebalance,\n <0.10041.15>,\n {dcp_takeover,\n \'ns_1@172.23.122.150\',\n 514}},\n infinity]}}}}'} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661524823L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:58:44.823Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661523501L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:58:43.501Z', u'text': u'Started rebalancing bucket default'} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661523403L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:58:43.403Z', u'text': u'Bucket "bucket2" rebalance appears to be swap rebalance'} 2023-04-27 22:59:21,608 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661522888L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:58:42.888Z', u'text': u'Started rebalancing bucket bucket2'} 2023-04-27 22:59:21,609 | test | ERROR | MainThread | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661522774L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:58:42.774Z', u'text': u'Bucket "bucket1" rebalance appears to be swap rebalance'} ERROR 2023-04-27 22:59:21,651 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ | 172.23.122.162 | kv | 93.4000499277 | 11.74 GiB | 10.09 GiB | 512.00 KiB / 4.10 GiB | 2837071 / 1241936 | 7.2.0-5318-enterprise | | 172.23.122.141 | kv | 83.1057032097 | 11.74 GiB | 10.39 GiB | 0.0 Byte / 4.10 GiB | 3410983 / 4654907 | 7.2.0-5318-enterprise | | 172.23.122.161 | kv | 92.6898048585 | 11.74 GiB | 10.11 GiB | 58.50 MiB / 4.10 GiB | 2957828 / 1101589 | 7.2.0-5318-enterprise | | 172.23.122.150 | kv | 93.0255427209 | 11.74 GiB | 9.37 GiB | 0.0 Byte / 0.0 Byte | 3314420 / 4028515 | 7.2.0-5318-enterprise | | 172.23.122.139 | kv | 96.7313650901 | 11.74 GiB | 9.49 GiB | 0.0 Byte / 4.10 GiB | 3646783 / 4858113 | 7.2.0-5318-enterprise | | 172.23.122.163 | kv | 93.0713943341 | 11.74 GiB | 9.49 GiB | 0.0 Byte / 0.0 Byte | 3507141 / 3911007 | 7.2.0-5318-enterprise | +----------------+----------+-----------------+-----------+-----------+-----------------------+-------------------+-----------------------+ 2023-04-27 22:59:22,960 | test | ERROR | pool-8-thread-12 | [rest_client:_rebalance_status_and_progress:1556] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0922e99a89c59a425f3a98944f44d1ac', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=3ea5652b0c733b191fbc2da02a7777c4', u'status': u'notRunning'} - rebalance failed 2023-04-27 22:59:22,987 | test | INFO | pool-8-thread-12 | [rest_client:print_UI_logs:2738] Latest logs from UI on 172.23.122.150: 2023-04-27 22:59:22,987 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.122.139', u'tstamp': 1682661561709L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.709Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2023-04-27 22:59:22,989 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.122.163', u'tstamp': 1682661561698L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.698Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2023-04-27 22:59:22,989 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.122.162', u'tstamp': 1682661561698L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.698Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2023-04-27 22:59:22,989 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661561697L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.697Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2023-04-27 22:59:22,989 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.122.161', u'tstamp': 1682661561692L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.692Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2023-04-27 22:59:22,990 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661561687L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.687Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2023-04-27 22:59:22,990 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661561238L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:21.238Z', u'text': u'Bucket "bucket1" loaded on node \'ns_1@172.23.122.141\' in 0 seconds.'} 2023-04-27 22:59:22,990 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661559689L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.689Z', u'text': u"Service 'memcached' exited with status 139. Restarting. Messages:\n #9 /opt/couchbase/bin/memcached() [0x400000+0x23b83b]\n #10 /opt/couchbase/bin/memcached() [0x400000+0x20e0cb]\n #11 /opt/couchbase/bin/memcached() [0x400000+0x20e160]\n #12 /opt/couchbase/bin/memcached() [0x400000+0x1ef543]\n #13 /opt/couchbase/bin/memcached() [0x400000+0x1ef7f2]\n #14 /opt/couchbase/bin/memcached() [0x400000+0x1efe7b]\n #15 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f9a4e093000+0xed7e]\n #16 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f9a4e093000+0x17cf1]\n #17 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7f9a4e093000+0x1853f]\n #18 /opt/couchbase/bin/memcached() [0x400000+0x824927]\n #19 /opt/couchbase/bin/memcached() [0x400000+0x824df6]\n #20 /opt/couchbase/bin/memcached() [0x400000+0x8267b6]\n #21 /opt/couchbase/bin/memcached() [0x400000+0x1bbfc9]\n #22 /opt/couchbase/bin/memcached() [0x400000+0x7a9409]\n #23 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f9a4ebd3000+0x7fa3]\n #24 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f9a4c9aa000+0xf8eff]"} 2023-04-27 22:59:22,990 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.122.150', u'tstamp': 1682661559644L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.644Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.23052.15>,\n {{{{{child_interrupted,\n {\'EXIT\',<26289.7918.4>,socket_closed}},\n [{dcp_replicator,spawn_and_wait,1,\n [{file,"src/dcp_replicator.erl"},\n {line,358}]},\n {dcp_replicator,handle_call,3,\n [{file,"src/dcp_replicator.erl"},\n {line,152}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,721}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,750}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [<26289.7916.4>,get_partitions,infinity]}},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,63},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.122.162\'},\n {if_rebalance,<0.10041.15>,\n {dcp_takeover,\'ns_1@172.23.122.150\',514}},\n infinity]}}}}}.\nRebalance Operation Id = e98e17eddc3d7b874e5da9ee8fbccfa8'} 2023-04-27 22:59:22,990 | test | ERROR | pool-8-thread-12 | [rest_client:print_UI_logs:2740] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.122.141', u'tstamp': 1682661559617L, u'shortText': u'message', u'serverTime': u'2023-04-27T22:59:19.617Z', u'text': u"Control connection to memcached on 'ns_1@172.23.122.141' disconnected. Check logs for details."} 2023-04-27 22:59:22,993 | test | ERROR | pool-8-thread-12 | [task:call:332] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0922e99a89c59a425f3a98944f44d1ac', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=3ea5652b0c733b191fbc2da02a7777c4', u'status': u'notRunning'} - rebalance failed 2023-04-27 22:59:23,526 | test | ERROR | pool-8-thread-20 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 20 docs from 500 to 520 of thread_name LoadDocs_MutateDocsFromSpecTask_316771_bucket1__default_collection-1000_cont_update_ttl=0_update__500_520_316774 ############################## 172.23.122.141: 1 core dump seen running: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/d83a36fe-cbc1-4e17-5455e38b-112a8819.dmp > /opt/couchbase/var/lib/couchbase/crash/83a36fe-cbc1-4e17-5455e38b-112a8819.core running: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/83a36fe-cbc1-4e17-5455e38b-112a8819.core -ex "bt full" -ex quit 172.23.122.141: Stack Trace of first crash - d83a36fe-cbc1-4e17-5455e38b-112a8819.dmp ############################## running: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2023-04-27 22:59:26,098 | test | INFO | MainThread | [basetestcase:check_logs:813] Comparing timestamps: Log's latest timestamp: 2023-04-27 22:59:19, Test's start timestamp is 2023-04-27 22:30:06.974000 2023-04-27 22:59:26,101 | test | INFO | MainThread | [basetestcase:check_logs:813] Comparing timestamps: Log's latest timestamp: 2023-04-27 22:59:19, Test's start timestamp is 2023-04-27 22:30:06.974000 2023-04-27 22:59:26,101 | test | INFO | MainThread | [basetestcase:check_coredump_exist:922] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2023-04-27 22:59:26,101 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:924] 172.23.122.141: Found ' CRITICAL ' logs - ['[ns_server:info,2023-04-27T22:59:19.653-07:00,babysitter_of_ns_1@cb.local:<0.135.0>:ns_port_server:log:226]memcached<0.135.0>: CRITICAL Breakpad caught a crash (Couchbase version 7.2.0-5318). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/d83a36fe-cbc1-4e17-5455e38b-112a8819.dmp before terminating.\n', '[ns_server:info,2023-04-27T22:59:19.981-07:00,babysitter_of_ns_1@cb.local:<0.851.0>:ns_port_server:log:226]memcached<0.851.0>: 2023-04-27T22:59:19.780913-07:00 CRITICAL Detected previous crash\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781008-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.2.0-5318). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/d83a36fe-cbc1-4e17-5455e38b-112a8819.dmp before terminating.\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781017-07:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781019-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x7431f8]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781021-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x794cea]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781023-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x795028]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781050-07:00 CRITICAL #3 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f9a4ebd3000+0x12730]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781055-07:00 CRITICAL #4 /lib/x86_64-linux-gnu/libpthread.so.0(__pthread_mutex_lock+0) [0x7f9a4ebd3000+0xa6c0]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781058-07:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x4fe128]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781060-07:00 CRITICAL #6 /opt/couchbase/bin/memcached() [0x400000+0x4f7a54]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781065-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x2b543b]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781067-07:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x198f3f]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781122-07:00 CRITICAL #9 /opt/couchbase/bin/memcached() [0x400000+0x23b83b]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781124-07:00 CRITICAL #10 /opt/couchbase/bin/memcached() [0x400000+0x20e0cb]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781126-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x20e160]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781128-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x1ef543]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781150-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x1ef7f2]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781152-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x1efe7b]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781157-07:00 CRITICAL #15 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f9a4e093000+0xed7e]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781160-07:00 CRITICAL #16 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7() [0x7f9a4e093000+0x17cf1]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781165-07:00 CRITICAL #17 /opt/couchbase/bin/../lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7f9a4e093000+0x1853f]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781167-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x824927]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781189-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x824df6]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781210-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x8267b6]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781232-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x1bbfc9]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781234-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x7a9409]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781239-07:00 CRITICAL #23 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f9a4ebd3000+0x7fa3]\n', 'memcached<0.851.0>: 2023-04-27T22:59:19.781242-07:00 CRITICAL #24 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f9a4c9aa000+0xf8eff]\n'] 2023-04-27 22:59:27,493 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1214] Running cbcollect on node ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 2023-04-27 22:59:37,513 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1217] ns_1@172.23.122.139,ns_1@172.23.122.141,ns_1@172.23.122.150,ns_1@172.23.122.161,ns_1@172.23.122.162,ns_1@172.23.122.163 - cbcollect status: True 2023-04-27 22:59:37,513 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1222] Polling active_tasks to check cbcollect status 2023-04-27 23:00:23,473 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 617 docs from 37183 to 39140 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__37183_39140_324390 2023-04-27 23:00:23,473 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 2 docs from 39140 to 39150 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-77_collection-1258_update_ttl=0_update__39140_39150_324392 2023-04-27 23:00:23,677 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 775 docs from 0 to 1957 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__0_1957_324394 2023-04-27 23:00:23,677 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 604 docs from 1957 to 3914 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__1957_3914_324396 2023-04-27 23:00:23,697 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 625 docs from 3914 to 5871 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__3914_5871_324398 2023-04-27 23:00:23,698 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 598 docs from 5871 to 7828 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__5871_7828_324400 2023-04-27 23:00:23,700 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 640 docs from 7828 to 9785 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__7828_9785_324402 2023-04-27 23:00:23,713 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 752 docs from 9785 to 11742 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__9785_11742_324404 2023-04-27 23:00:23,714 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 734 docs from 11742 to 13699 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__11742_13699_324406 2023-04-27 23:00:23,716 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 734 docs from 13699 to 15656 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__13699_15656_324408 2023-04-27 23:00:23,717 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 767 docs from 15656 to 17613 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__15656_17613_324410 2023-04-27 23:00:23,717 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 734 docs from 17613 to 19570 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__17613_19570_324412 2023-04-27 23:00:23,717 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 755 docs from 19570 to 21527 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__19570_21527_324414 2023-04-27 23:00:23,717 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 761 docs from 21527 to 23484 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__21527_23484_324416 2023-04-27 23:00:23,719 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 700 docs from 23484 to 25441 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__23484_25441_324418 2023-04-27 23:00:23,719 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 728 docs from 25441 to 27398 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__25441_27398_324420 2023-04-27 23:00:23,720 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 744 docs from 27398 to 29355 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__27398_29355_324422 2023-04-27 23:00:23,720 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 709 docs from 29355 to 31312 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__29355_31312_324424 2023-04-27 23:00:23,720 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 641 docs from 31312 to 33269 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__31312_33269_324426 2023-04-27 23:00:23,721 | test | ERROR | pool-8-thread-23 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 1275 docs from 33269 to 35226 of thread_name LoadDocs_MutateDocsFromSpecTask_316770_default_scope-75_collection-1197_update_ttl=0_update__33269_35226_324428 2023-04-27 23:02:53,664 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.139: Copying cbcollect ZIP file to Client 2023-04-27 23:02:54,165 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.141: Copying cbcollect ZIP file to Client 2023-04-27 23:02:54,762 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.150: Copying cbcollect ZIP file to Client 2023-04-27 23:02:55,410 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.161: Copying cbcollect ZIP file to Client 2023-04-27 23:02:55,961 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.162: Copying cbcollect ZIP file to Client 2023-04-27 23:02:56,493 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1256] ns_1@172.23.122.163: Copying cbcollect ZIP file to Client 2023-04-27 23:02:57,115 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:586] Skipping get_trace !! ERROR During the test, Remote Connections: 294, Disconnections: 294 SDK Connections: 30, Disconnections: 30 ====================================================================== ERROR: test_start_stop_rebalance_with_mutations (rebalance_new.rebalance_start_stop.RebalanceStartStopTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/rebalance_start_stop.py", line 180, in test_start_stop_rebalance_with_mutations reached = self.cluster_util.rebalance_reached( File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 845, in rebalance_reached progress = rest._rebalance_progress() File "lib/membase/api/rest_client.py", line 1596, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1558, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0922e99a89c59a425f3a98944f44d1ac', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=3ea5652b0c733b191fbc2da02a7777c4', u'status': u'notRunning'} - rebalance failed ====================================================================== ERROR: test_start_stop_rebalance_with_mutations (rebalance_new.rebalance_start_stop.RebalanceStartStopTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance_new/rebalance_start_stop.py", line 41, in tearDown super(RebalanceStartStopTests, self).tearDown() File "pytests/rebalance_new/rebalance_base.py", line 216, in tearDown super(RebalanceBaseTest, self).tearDown() File "pytests/basetestcase.py", line 548, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 1970.371s