[2022-03-21 06:23:23,726] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:23:23,835] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:23:24,105] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:24,410] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:24,415] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:23:24,424] - [rest_client] [139784139831040] - INFO - Node version in cluster 7.1.0-2506-enterprise [2022-03-21 06:23:24,425] - [basetestcase] [139784139831040] - INFO - ============== basetestcase setup was started for test #2 test_rebalance_out_of_nodes_with_failed_rebalance============== [2022-03-21 06:23:24,425] - [basetestcase] [139784139831040] - INFO - Building docker image with java sdk client [2022-03-21 06:23:28,180] - [basetestcase] [139784139831040] - INFO - initializing cluster [2022-03-21 06:23:28,734] - [task] [139784097867520] - INFO - server: ip:172.23.97.145 port:8091 ssh_username:root, nodes/self [2022-03-21 06:23:28,738] - [task] [139784097867520] - INFO - {'uptime': '1043', 'memoryTotal': 3935289344, 'memoryFree': 2823245824, 'mcdMemoryReserved': 3002, 'mcdMemoryAllocated': 3002, 'status': 'healthy', 'hostname': '172.23.97.145:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1006, 'memcached': 11210, 'id': 'ns_1@172.23.97.145', 'ip': '172.23.97.145', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 3752, 'curr_items': 0} [2022-03-21 06:23:28,738] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1005 [2022-03-21 06:23:28,751] - [task] [139784097867520] - INFO - quota for index service will be 1005 MB [2022-03-21 06:23:28,751] - [task] [139784097867520] - INFO - set index quota to node 172.23.97.145 [2022-03-21 06:23:28,751] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1005 [2022-03-21 06:23:28,762] - [rest_client] [139784097867520] - INFO - pools/default params : memoryQuota=1006 [2022-03-21 06:23:28,766] - [rest_client] [139784097867520] - INFO - --> init_node_services(Administrator,password,172.23.97.145,8091,['kv', 'n1ql', 'index']) [2022-03-21 06:23:28,767] - [rest_client] [139784097867520] - INFO - /node/controller/setupServices params on 172.23.97.145: 8091:hostname=172.23.97.145&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex [2022-03-21 06:23:28,768] - [rest_client] [139784097867520] - ERROR - POST http://172.23.97.145:8091//node/controller/setupServices body: hostname=172.23.97.145&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-21 06:23:28,769] - [rest_client] [139784097867520] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-21 06:23:28,769] - [rest_client] [139784097867520] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:23:28,769] - [rest_client] [139784097867520] - INFO - settings/web params on 172.23.97.145:8091:port=8091&username=Administrator&password=password [2022-03-21 06:23:28,819] - [rest_client] [139784097867520] - INFO - --> status:True [2022-03-21 06:23:28,821] - [remote_util] [139784097867520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:23:28,931] - [remote_util] [139784097867520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:23:29,203] - [remote_util] [139784097867520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:29,510] - [remote_util] [139784097867520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:29,511] - [remote_util] [139784097867520] - INFO - running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:23:29,578] - [remote_util] [139784097867520] - INFO - command executed successfully with root [2022-03-21 06:23:29,578] - [remote_util] [139784097867520] - INFO - ['ok'] [2022-03-21 06:23:29,581] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.97.145:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:29,583] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.97.145:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:29,589] - [rest_client] [139784097867520] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:23:29,611] - [task] [139784097867520] - INFO - server: ip:172.23.106.184 port:8091 ssh_username:root, nodes/self [2022-03-21 06:23:29,617] - [task] [139784097867520] - INFO - {'uptime': '1029', 'memoryTotal': 3945775104, 'memoryFree': 3287773184, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.184:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.106.184', 'ip': '172.23.106.184', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:23:29,617] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:23:29,629] - [rest_client] [139784097867520] - ERROR - GET http://172.23.106.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-21 06:23:29,629] - [rest_client] [139784097867520] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:23:29,637] - [rest_client] [139784097867520] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:23:29,637] - [rest_client] [139784097867520] - INFO - settings/web params on 172.23.106.184:8091:port=8091&username=Administrator&password=password [2022-03-21 06:23:29,721] - [rest_client] [139784097867520] - INFO - --> status:True [2022-03-21 06:23:29,722] - [remote_util] [139784097867520] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:23:29,859] - [remote_util] [139784097867520] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:23:30,427] - [remote_util] [139784097867520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:30,862] - [remote_util] [139784097867520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:30,863] - [remote_util] [139784097867520] - INFO - running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:23:30,915] - [remote_util] [139784097867520] - INFO - command executed successfully with root [2022-03-21 06:23:30,915] - [remote_util] [139784097867520] - INFO - ['ok'] [2022-03-21 06:23:30,919] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.106.184:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:30,922] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.106.184:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:30,928] - [rest_client] [139784097867520] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:23:30,954] - [task] [139784097867520] - INFO - server: ip:172.23.107.235 port:8091 ssh_username:root, nodes/self [2022-03-21 06:23:30,960] - [task] [139784097867520] - INFO - {'uptime': '240', 'memoryTotal': 3945775104, 'memoryFree': 3266048000, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.235:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.107.235', 'ip': '172.23.107.235', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:23:30,960] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:23:30,969] - [rest_client] [139784097867520] - ERROR - GET http://172.23.107.235:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-21 06:23:30,970] - [rest_client] [139784097867520] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:23:30,975] - [rest_client] [139784097867520] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:23:30,975] - [rest_client] [139784097867520] - INFO - settings/web params on 172.23.107.235:8091:port=8091&username=Administrator&password=password [2022-03-21 06:23:31,053] - [rest_client] [139784097867520] - INFO - --> status:True [2022-03-21 06:23:31,054] - [remote_util] [139784097867520] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:23:31,198] - [remote_util] [139784097867520] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:23:31,573] - [remote_util] [139784097867520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:32,010] - [remote_util] [139784097867520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:32,010] - [remote_util] [139784097867520] - INFO - running command.raw on 172.23.107.235: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:23:32,103] - [remote_util] [139784097867520] - INFO - command executed successfully with root [2022-03-21 06:23:32,103] - [remote_util] [139784097867520] - INFO - ['ok'] [2022-03-21 06:23:32,107] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.235:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:32,111] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.235:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:32,118] - [rest_client] [139784097867520] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:23:32,142] - [task] [139784097867520] - INFO - server: ip:172.23.107.230 port:8091 ssh_username:root, nodes/self [2022-03-21 06:23:32,149] - [task] [139784097867520] - INFO - {'uptime': '1030', 'memoryTotal': 3945775104, 'memoryFree': 3268947968, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.230:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2016, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.230', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:23:32,149] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:23:32,163] - [rest_client] [139784097867520] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:23:32,169] - [rest_client] [139784097867520] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:23:32,169] - [rest_client] [139784097867520] - INFO - settings/web params on 172.23.107.230:8091:port=8091&username=Administrator&password=password [2022-03-21 06:23:32,264] - [rest_client] [139784097867520] - INFO - --> status:True [2022-03-21 06:23:32,265] - [remote_util] [139784097867520] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:23:32,439] - [remote_util] [139784097867520] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:23:32,851] - [remote_util] [139784097867520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:33,298] - [remote_util] [139784097867520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:33,299] - [remote_util] [139784097867520] - INFO - running command.raw on 172.23.107.230: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:23:33,390] - [remote_util] [139784097867520] - INFO - command executed successfully with root [2022-03-21 06:23:33,391] - [remote_util] [139784097867520] - INFO - ['ok'] [2022-03-21 06:23:33,395] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.230:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:33,398] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.230:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:33,405] - [rest_client] [139784097867520] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:23:33,428] - [task] [139784097867520] - INFO - server: ip:172.23.107.24 port:8091 ssh_username:root, nodes/self [2022-03-21 06:23:33,435] - [task] [139784097867520] - INFO - {'uptime': '1035', 'memoryTotal': 3945775104, 'memoryFree': 3299373056, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.24:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2016, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.24', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:23:33,436] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:23:33,450] - [rest_client] [139784097867520] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:23:33,456] - [rest_client] [139784097867520] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:23:33,456] - [rest_client] [139784097867520] - INFO - settings/web params on 172.23.107.24:8091:port=8091&username=Administrator&password=password [2022-03-21 06:23:33,515] - [rest_client] [139784097867520] - INFO - --> status:True [2022-03-21 06:23:33,517] - [remote_util] [139784097867520] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:23:33,680] - [remote_util] [139784097867520] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:23:34,080] - [remote_util] [139784097867520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:34,522] - [remote_util] [139784097867520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:34,523] - [remote_util] [139784097867520] - INFO - running command.raw on 172.23.107.24: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:23:34,576] - [remote_util] [139784097867520] - INFO - command executed successfully with root [2022-03-21 06:23:34,577] - [remote_util] [139784097867520] - INFO - ['ok'] [2022-03-21 06:23:34,580] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.24:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:34,583] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.24:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:34,590] - [rest_client] [139784097867520] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:23:34,610] - [task] [139784097867520] - INFO - server: ip:172.23.107.247 port:8091 ssh_username:root, nodes/self [2022-03-21 06:23:34,616] - [task] [139784097867520] - INFO - {'uptime': '1035', 'memoryTotal': 3945775104, 'memoryFree': 3292663808, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.247:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2016, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.247', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:23:34,616] - [rest_client] [139784097867520] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:23:34,632] - [rest_client] [139784097867520] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:23:34,638] - [rest_client] [139784097867520] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:23:34,639] - [rest_client] [139784097867520] - INFO - settings/web params on 172.23.107.247:8091:port=8091&username=Administrator&password=password [2022-03-21 06:23:34,699] - [rest_client] [139784097867520] - INFO - --> status:True [2022-03-21 06:23:34,700] - [remote_util] [139784097867520] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:23:34,849] - [remote_util] [139784097867520] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:23:35,231] - [remote_util] [139784097867520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:23:35,661] - [remote_util] [139784097867520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:23:35,662] - [remote_util] [139784097867520] - INFO - running command.raw on 172.23.107.247: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:23:35,752] - [remote_util] [139784097867520] - INFO - command executed successfully with root [2022-03-21 06:23:35,752] - [remote_util] [139784097867520] - INFO - ['ok'] [2022-03-21 06:23:35,756] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.247:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:35,759] - [rest_client] [139784097867520] - INFO - /diag/eval status on 172.23.107.247:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:23:35,765] - [rest_client] [139784097867520] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:23:35,779] - [basetestcase] [139784139831040] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.97.145 **** [2022-03-21 06:23:35,865] - [basetestcase] [139784139831040] - INFO - sleep for 5 secs. ... [2022-03-21 06:23:40,871] - [basetestcase] [139784139831040] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-21 06:23:40,882] - [basetestcase] [139784139831040] - INFO - done initializing cluster [2022-03-21 06:23:41,786] - [task] [139784097867520] - INFO - adding node 172.23.106.184:8091 to cluster [2022-03-21 06:23:41,793] - [rest_client] [139784097867520] - INFO - adding remote node @172.23.106.184:18091 to this cluster @172.23.97.145:8091 [2022-03-21 06:23:51,805] - [rest_client] [139784097867520] - INFO - rebalance progress took 10.01 seconds [2022-03-21 06:23:51,805] - [rest_client] [139784097867520] - INFO - sleep for 10 seconds after rebalance... [2022-03-21 06:24:05,881] - [task] [139784097867520] - INFO - adding node 172.23.107.235:8091 to cluster [2022-03-21 06:24:05,890] - [rest_client] [139784097867520] - INFO - adding remote node @172.23.107.235:18091 to this cluster @172.23.97.145:8091 [2022-03-21 06:24:15,904] - [rest_client] [139784097867520] - INFO - rebalance progress took 10.01 seconds [2022-03-21 06:24:15,904] - [rest_client] [139784097867520] - INFO - sleep for 10 seconds after rebalance... [2022-03-21 06:24:29,958] - [rest_client] [139784097867520] - INFO - Node 172.23.106.184 not part of cluster inactiveAdded [2022-03-21 06:24:29,959] - [rest_client] [139784097867520] - INFO - Node 172.23.107.235 not part of cluster inactiveAdded [2022-03-21 06:24:29,964] - [rest_client] [139784097867520] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235,ns_1@172.23.97.145', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-21 06:24:30,017] - [rest_client] [139784097867520] - INFO - rebalance operation started [2022-03-21 06:24:40,030] - [task] [139784097867520] - INFO - Rebalance - status: none, progress: 100.00% [2022-03-21 06:24:40,035] - [task] [139784097867520] - INFO - rebalancing was completed with progress: 100% in 10.018119812011719 sec [2022-03-21 06:24:40,037] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:24:40,138] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:24:40,404] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:40,699] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:40,700] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:24:40,763] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:24:40,763] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:24:40,763] - [basetestcase] [139784139831040] - INFO - Enabled diag/eval for non-local hosts from 172.23.97.145 [2022-03-21 06:24:40,763] - [basetestcase] [139784139831040] - INFO - ============== basetestcase setup was finished for test #2 test_rebalance_out_of_nodes_with_failed_rebalance ============== [2022-03-21 06:24:40,770] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:24:40,867] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:24:41,128] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:41,423] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:41,426] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:24:41,601] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:24:41,980] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:42,419] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:42,422] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:24:42,586] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:24:42,974] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:43,426] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:43,428] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:24:43,597] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:24:44,003] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:44,454] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:44,456] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:24:44,615] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:24:44,975] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:45,417] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:45,419] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:24:45,582] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:24:45,988] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:24:46,425] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:24:56,940] - [basetestcase] [139784139831040] - INFO - ------- Cluster statistics ------- [2022-03-21 06:24:56,940] - [basetestcase] [139784139831040] - INFO - 172.23.106.184:8091 => {'services': ['index'], 'cpu_utilization': 12.15567349001769, 'mem_free': 3266527232, 'mem_total': 3945775104, 'swap_mem_used': 44564480, 'swap_mem_total': 3758092288} [2022-03-21 06:24:56,940] - [basetestcase] [139784139831040] - INFO - 172.23.107.235:8091 => {'services': ['index'], 'cpu_utilization': 3.761676344357486, 'mem_free': 3228721152, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:24:56,940] - [basetestcase] [139784139831040] - INFO - 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 8.037288989669943, 'mem_free': 2822701056, 'mem_total': 3935289344, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:24:56,940] - [basetestcase] [139784139831040] - INFO - --- End of cluster statistics --- [2022-03-21 06:24:56,940] - [basetestcase] [139784139831040] - INFO - Time to execute basesetup : 93.21543312072754 [2022-03-21 06:24:56,946] - [newtuq] [139784139831040] - INFO - Initial status of 172.23.97.145 cluster is healthy [2022-03-21 06:24:56,947] - [newtuq] [139784139831040] - INFO - current status of 172.23.97.145 is healthy [2022-03-21 06:24:56,968] - [newtuq] [139784139831040] - INFO - Initial status of 172.23.106.184 cluster is healthy [2022-03-21 06:24:56,968] - [newtuq] [139784139831040] - INFO - current status of 172.23.106.184 is healthy [2022-03-21 06:24:56,988] - [newtuq] [139784139831040] - INFO - Initial status of 172.23.107.235 cluster is healthy [2022-03-21 06:24:56,989] - [newtuq] [139784139831040] - INFO - current status of 172.23.107.235 is healthy [2022-03-21 06:24:57,002] - [newtuq] [139784139831040] - INFO - Initial status of 172.23.107.230 cluster is healthy [2022-03-21 06:24:57,002] - [newtuq] [139784139831040] - INFO - current status of 172.23.107.230 is healthy [2022-03-21 06:24:57,015] - [newtuq] [139784139831040] - INFO - Initial status of 172.23.107.24 cluster is healthy [2022-03-21 06:24:57,015] - [newtuq] [139784139831040] - INFO - current status of 172.23.107.24 is healthy [2022-03-21 06:24:57,028] - [newtuq] [139784139831040] - INFO - Initial status of 172.23.107.247 cluster is healthy [2022-03-21 06:24:57,028] - [newtuq] [139784139831040] - INFO - current status of 172.23.107.247 is healthy [2022-03-21 06:24:57,034] - [basetestcase] [139784139831040] - INFO - list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.235 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:24:57,059] - [rest_client] [139784139831040] - INFO - {'indexer.settings.storage_mode': 'plasma'} set [2022-03-21 06:24:57,059] - [newtuq] [139784139831040] - INFO - Allowing the indexer to complete restart after setting the internal settings [2022-03-21 06:24:57,059] - [basetestcase] [139784139831040] - INFO - sleep for 5 secs. ... [2022-03-21 06:25:02,077] - [rest_client] [139784139831040] - INFO - {'indexer.api.enableTestServer': True} set [2022-03-21 06:25:02,078] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:25:02,180] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:25:02,445] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:25:02,748] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:25:09,578] - [basetestcase] [139784139831040] - INFO - LOAD IS FINISHED [2022-03-21 06:25:09,587] - [basetestcase] [139784139831040] - INFO - list of n1ql nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:25:09,587] - [newtuq] [139784139831040] - INFO - ip:172.23.97.145 port:8091 ssh_username:root [2022-03-21 06:25:09,588] - [basetestcase] [139784139831040] - INFO - sleep for 30 secs. ... [2022-03-21 06:25:39,627] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:25:39,731] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:25:39,998] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:25:40,300] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:25:40,315] - [basetestcase] [139784139831040] - INFO - list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.235 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:25:40,324] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.106.184:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False [2022-03-21 06:25:40,340] - [rest_client] [139784139831040] - INFO - {'queryport.client.waitForScheduledIndex': False} set [2022-03-21 06:25:40,340] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.106.184:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False [2022-03-21 06:25:40,355] - [rest_client] [139784139831040] - INFO - {'indexer.allowScheduleCreateRebal': True} set [2022-03-21 06:25:40,355] - [collections_indexes_rebalance] [139784139831040] - INFO - ============== ConcurrentIndexes setup has started ============== [2022-03-21 06:25:41,027] - [rest_client] [139784097867520] - INFO - http://172.23.97.145:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=256&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-21 06:25:41,057] - [rest_client] [139784097867520] - INFO - 0.03 seconds to create bucket test_bucket [2022-03-21 06:25:41,057] - [bucket_helper] [139784097867520] - INFO - waiting for memcached bucket : test_bucket in 172.23.97.145 to accept set ops [2022-03-21 06:25:41,630] - [data_helper] [139784097867520] - INFO - creating direct client 172.23.97.145:11210 test_bucket [2022-03-21 06:25:42,004] - [data_helper] [139784097867520] - INFO - creating direct client 172.23.97.145:11210 test_bucket [2022-03-21 06:25:42,981] - [task] [139784097867520] - INFO - bucket 'test_bucket' was created with per node RAM quota: 256 [2022-03-21 06:25:42,996] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:25:43,095] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:25:43,359] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:25:43,659] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:25:43,660] - [collections_indexes_rebalance] [139784139831040] - INFO - ============== ConcurrentIndexes setup has completed ============== [2022-03-21 06:25:43,695] - [rest_client] [139784139831040] - INFO - http://172.23.97.145:8091/pools/default/buckets/test_bucket/scopes with params: name=test_scope_1 [2022-03-21 06:25:43,696] - [rest_client] [139784139831040] - INFO - Scope created test_bucket->test_scope_1 {'uid': '1'} [2022-03-21 06:25:43,750] - [rest_client] [139784139831040] - INFO - http://172.23.97.145:8091/pools/default/buckets/test_bucket/scopes/test_scope_1/collections with params: name=test_collection_1 [2022-03-21 06:25:43,751] - [rest_client] [139784139831040] - INFO - Collection created test_bucket->test_scope_1->test_collection_1 manifest:{'uid': '2'} [2022-03-21 06:25:43,751] - [basetestcase] [139784139831040] - INFO - sleep for 10 secs. Allowing time after collection creation ... [2022-03-21 06:25:53,762] - [basetestcase] [139784139831040] - INFO - Number of batches : 10 [2022-03-21 06:25:53,762] - [basetestcase] [139784139831040] - INFO - sleep for 10 secs. ... [2022-03-21 06:26:03,772] - [basetestcase] [139784139831040] - INFO - done [2022-03-21 06:26:04,005] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:26:39,240] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 10001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:27:14,065] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 20001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:27:49,189] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 30001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:28:24,099] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 40001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:28:59,021] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 50001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:29:33,993] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 60001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:30:09,096] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 70001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:30:44,010] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 80001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:31:18,909] - [task] [139784097867520] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.97.145 -u Administrator -p password -b test_bucket -s test_scope_1 -c test_collection_1 -n 10000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 90001 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 99999 -o True -sd False --secure False [2022-03-21 06:31:24,236] - [tuq_helper] [139782966736640] - INFO - RUN QUERY CREATE INDEX `idx_0` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,237] - [tuq_helper] [139782924773120] - INFO - RUN QUERY CREATE INDEX `idx_5` ON default:test_bucket.test_scope_1.test_collection_1(lastName) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,238] - [tuq_helper] [139782958343936] - INFO - RUN QUERY CREATE INDEX `idx_1` ON default:test_bucket.test_scope_1.test_collection_1(city) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,240] - [tuq_helper] [139782941558528] - INFO - RUN QUERY CREATE INDEX `idx_3` ON default:test_bucket.test_scope_1.test_collection_1(title) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,241] - [tuq_helper] [139782916380416] - INFO - RUN QUERY CREATE INDEX `idx_6` ON default:test_bucket.test_scope_1.test_collection_1(streetAddress) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,242] - [tuq_helper] [139782899595008] - INFO - RUN QUERY CREATE INDEX `idx_8` ON default:test_bucket.test_scope_1.test_collection_1(filler1) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,243] - [tuq_helper] [139782949951232] - INFO - RUN QUERY CREATE INDEX `idx_2` ON default:test_bucket.test_scope_1.test_collection_1(country) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,244] - [rest_client] [139782958343936] - INFO - query params : statement=CREATE+INDEX+%60idx_1%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,244] - [tuq_helper] [139782907987712] - INFO - RUN QUERY CREATE INDEX `idx_7` ON default:test_bucket.test_scope_1.test_collection_1(suffix) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,245] - [tuq_helper] [139782933165824] - INFO - RUN QUERY CREATE INDEX `idx_4` ON default:test_bucket.test_scope_1.test_collection_1(firstName) USING GSI WITH {'defer_build': False, 'num_replica': 1} [2022-03-21 06:31:24,246] - [rest_client] [139782966736640] - INFO - query params : statement=CREATE+INDEX+%60idx_0%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,248] - [rest_client] [139782916380416] - INFO - query params : statement=CREATE+INDEX+%60idx_6%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28streetAddress%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,250] - [rest_client] [139782899595008] - INFO - query params : statement=CREATE+INDEX+%60idx_8%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28filler1%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,250] - [rest_client] [139782941558528] - INFO - query params : statement=CREATE+INDEX+%60idx_3%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,251] - [rest_client] [139782924773120] - INFO - query params : statement=CREATE+INDEX+%60idx_5%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28lastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,252] - [rest_client] [139782949951232] - INFO - query params : statement=CREATE+INDEX+%60idx_2%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,254] - [rest_client] [139782907987712] - INFO - query params : statement=CREATE+INDEX+%60idx_7%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28suffix%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,255] - [rest_client] [139782933165824] - INFO - query params : statement=CREATE+INDEX+%60idx_4%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D [2022-03-21 06:31:24,286] - [rest_client] [139782924773120] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_5%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28lastName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "352a7f3b-6d59-49d9-8b2f-a3971d84b7de",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_5, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "27.192799ms","executionTime": "27.159424ms","resultCount": 0,"resultSize": 0,"serviceLoad": 56,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:24,288] - [rest_client] [139782907987712] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_7%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28suffix%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "f9a97790-1c2e-4553-bc25-df7f69281932",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_7, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "29.259714ms","executionTime": "29.206117ms","resultCount": 0,"resultSize": 0,"serviceLoad": 50,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:24,301] - [rest_client] [139782949951232] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_2%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28country%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "ee7ec313-2b70-4919-9f64-a7dcf5e24add",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_2, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "42.524124ms","executionTime": "42.470957ms","resultCount": 0,"resultSize": 0,"serviceLoad": 43,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:24,316] - [rest_client] [139782933165824] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_4%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28firstName%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "fd72c090-2238-4321-bd20-9024ced2b024",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_4, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "56.264709ms","executionTime": "56.213815ms","resultCount": 0,"resultSize": 0,"serviceLoad": 37,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:24,317] - [rest_client] [139782916380416] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_6%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28streetAddress%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "6ed22f10-75a4-42e3-b22f-77413ebb9b6c",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_6, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "61.034928ms","executionTime": "60.992542ms","resultCount": 0,"resultSize": 0,"serviceLoad": 31,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:27,294] - [rest_client] [139782958343936] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_1%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28city%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "c43385fb-f78a-4568-974b-877aac10a265",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_1, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "3.042694952s","executionTime": "3.042630663s","resultCount": 0,"resultSize": 0,"serviceLoad": 25,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:27,301] - [rest_client] [139782941558528] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_3%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "24e1a7f8-2895-407d-a3fe-23a21b043802",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_3, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "3.043420559s","executionTime": "3.04338393s","resultCount": 0,"resultSize": 0,"serviceLoad": 18,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:27,303] - [rest_client] [139782966736640] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_0%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28age%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "ed5d4d5c-c46b-44c1-8b57-8805ca2fa524",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_0, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "3.048433546s","executionTime": "3.048361597s","resultCount": 0,"resultSize": 0,"serviceLoad": 12,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:27,304] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_0 is scheduled for background [2022-03-21 06:31:27,304] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_1 is scheduled for background [2022-03-21 06:31:27,304] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_2 is scheduled for background [2022-03-21 06:31:27,304] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_3 is scheduled for background [2022-03-21 06:31:27,304] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_4 is scheduled for background [2022-03-21 06:31:27,304] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_5 is scheduled for background [2022-03-21 06:31:27,305] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_6 is scheduled for background [2022-03-21 06:31:27,305] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_7 is scheduled for background [2022-03-21 06:31:27,306] - [rest_client] [139782899595008] - ERROR - POST http://172.23.97.145:8093/query?statement=CREATE+INDEX+%60idx_8%60+ON+default%3Atest_bucket.test_scope_1.test_collection_1%28filler1%29+USING+GSI++WITH+%7B%27defer_build%27%3A+False%2C+%27num_replica%27%3A+1%7D body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "f68759fe-2ea8-462a-b772-fe5c71270ed0",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: The index is scheduled for background creation. Index creation for index idx_8, bucket test_bucket, scope test_scope_1, collection test_collection_1 cannot start. Reason: Create index or Alter replica cannot proceed due to another concurrent create index request.."}],\n"status": "errors",\n"metrics": {"elapsedTime": "3.04987598s","executionTime": "3.049822188s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2022-03-21 06:31:27,306] - [collections_indexes_rebalance] [139784139831040] - INFO - idx_8 is scheduled for background [2022-03-21 06:31:27,311] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:28,317] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:29,323] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:30,329] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:31,335] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:32,343] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:33,349] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:34,355] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:35,366] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:36,380] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:37,393] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:38,403] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:39,410] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:40,418] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:41,424] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:42,430] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:43,439] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:44,451] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:45,459] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:46,466] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:47,472] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:48,486] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:49,493] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:50,499] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:51,509] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:52,518] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:53,536] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:54,543] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:55,553] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:56,559] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:57,565] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:58,572] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:31:59,580] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:00,587] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:01,595] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:02,604] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:03,612] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:04,619] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:05,626] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:06,633] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:07,646] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:08,652] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:09,659] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:10,669] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:11,675] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:12,682] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:13,688] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:14,694] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:15,709] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:16,721] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:17,733] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:18,741] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:19,750] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:20,763] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:21,769] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:22,778] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:23,793] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:24,801] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:25,819] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:26,825] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:27,834] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:28,840] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:29,846] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:30,853] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:31,861] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:32,867] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:32,874] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:9102/getIndexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:32,889] - [collections_indexes_rebalance] [139784139831040] - INFO - Starting Rebalance out process [2022-03-21 06:32:32,894] - [basetestcase] [139784139831040] - INFO - sleep for 5 secs. Allowing some time for rebalance to make progress ... [2022-03-21 06:32:33,349] - [rest_client] [139784097867520] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235,ns_1@172.23.97.145', 'ejectedNodes': 'ns_1@172.23.107.235', 'user': 'Administrator', 'password': 'password'} [2022-03-21 06:32:33,400] - [rest_client] [139784097867520] - INFO - rebalance operation started [2022-03-21 06:32:37,903] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:32:38,117] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:32:38,484] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:32:38,924] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:32:38,968] - [remote_util] [139784139831040] - INFO - 172.23.107.235 **** The linux version file /opt/couchbase/ VERSION.txt exists [2022-03-21 06:32:39,094] - [remote_util] [139784139831040] - INFO - Running systemd command on this server [2022-03-21 06:32:39,094] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.235: systemctl stop couchbase-server.service [2022-03-21 06:32:41,380] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:32:41,380] - [basetestcase] [139784139831040] - INFO - Couchbase stopped [2022-03-21 06:32:41,381] - [basetestcase] [139784139831040] - INFO - sleep for 5 secs. ... [2022-03-21 06:32:43,413] - [rest_client] [139784097867520] - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2022-03-21 06:32:43,418] - [rest_client] [139784097867520] - INFO - Latest logs from UI on 172.23.97.145: [2022-03-21 06:32:43,418] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.106.184', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1647869560182, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.106.184' saw that node 'ns_1@172.23.107.235' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-21T06:32:40.182Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1647869560177, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.97.145' saw that node 'ns_1@172.23.107.235' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-21T06:32:40.177Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1647869559665, 'shortText': 'message', 'text': "Rebalance exited with reason {service_rebalance_failed,index,\n {agent_died,<34218.13277.0>,\n {lost_connection,\n {'ns_1@172.23.107.235',shutdown}}}}.\nRebalance Operation Id = f1f1000d922fe463c3f10862a5cdc580", 'serverTime': '2022-03-21T06:32:39.665Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 0, 'module': 'ns_vbucket_mover', 'tstamp': 1647869553690, 'shortText': 'message', 'text': 'Bucket "test_bucket" rebalance appears to be swap rebalance', 'serverTime': '2022-03-21T06:32:33.690Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 0, 'module': 'ns_rebalancer', 'tstamp': 1647869553568, 'shortText': 'message', 'text': 'Started rebalancing bucket test_bucket', 'serverTime': '2022-03-21T06:32:33.568Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1647869553351, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.106.184','ns_1@172.23.97.145'], EjectNodes = ['ns_1@172.23.107.235'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f1f1000d922fe463c3f10862a5cdc580", 'serverTime': '2022-03-21T06:32:33.351Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1647869141599, 'shortText': 'message', 'text': 'Bucket "test_bucket" loaded on node \'ns_1@172.23.97.145\' in 0 seconds.', 'serverTime': '2022-03-21T06:25:41.599Z'} [2022-03-21 06:32:43,419] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 12, 'module': 'menelaus_web', 'tstamp': 1647869141056, 'shortText': 'message', 'text': 'Created bucket "test_bucket" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,268435456},\n {durability_min_level,none},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', 'serverTime': '2022-03-21T06:25:41.056Z'} [2022-03-21 06:32:43,420] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1647869080768, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@172.23.97.145\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\n2022-03-21 06:24:40.764029 : test_rebalance_out_of_nodes_with_failed_rebalance started ', 'serverTime': '2022-03-21T06:24:40.768Z'} [2022-03-21 06:32:43,420] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1647869071588, 'shortText': 'message', 'text': 'Rebalance completed successfully.\nRebalance Operation Id = 375ffae22d8ac93bb37484d8c521746d', 'serverTime': '2022-03-21T06:24:31.588Z'} [2022-03-21 06:32:46,387] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:32:46,554] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:32:46,921] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:32:47,357] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:32:47,403] - [remote_util] [139784139831040] - INFO - 172.23.107.235 **** The linux version file /opt/couchbase/ VERSION.txt exists [2022-03-21 06:32:47,403] - [remote_util] [139784139831040] - INFO - Checking for process beam.smp on linux [2022-03-21 06:32:47,486] - [remote_util] [139784139831040] - INFO - Starting couchbase server [2022-03-21 06:32:47,614] - [remote_util] [139784139831040] - INFO - Running systemd command on this server [2022-03-21 06:32:47,615] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.235: systemctl start couchbase-server.service [2022-03-21 06:32:47,668] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:32:47,669] - [remote_util] [139784139831040] - INFO - 172.23.107.235:sleep for 5 secs. waiting for couchbase server to come up ... [2022-03-21 06:32:52,674] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.235: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2022-03-21 06:32:52,730] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:32:52,730] - [remote_util] [139784139831040] - INFO - Couchbase server status: [' Process: 77159 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2022-03-21 06:32:52,730] - [remote_util] [139784139831040] - INFO - Checking for process beam.smp on linux [2022-03-21 06:32:52,832] - [remote_util] [139784139831040] - INFO - process beam.smp is running on 172.23.107.235: with pid 77290 [2022-03-21 06:32:52,832] - [basetestcase] [139784139831040] - INFO - Couchbase started [2022-03-21 06:32:52,833] - [collections_indexes_rebalance] [139784139831040] - INFO - Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2022-03-21 06:32:52,837] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:8091/indexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:52,841] - [rest_client] [139784139831040] - INFO - Making a rest request api=http://172.23.97.145:9102/getIndexStatus verb=GET params= client_cert=None verify=False [2022-03-21 06:32:52,861] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where age is not null [2022-03-21 06:32:52,864] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+age+is+not+null [2022-03-21 06:32:52,871] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 4.965031ms [2022-03-21 06:32:52,874] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where city is not null [2022-03-21 06:32:52,878] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+city+is+not+null [2022-03-21 06:32:52,883] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 3.523933ms [2022-03-21 06:32:52,886] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where country is not null [2022-03-21 06:32:52,889] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+country+is+not+null [2022-03-21 06:32:52,893] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 1.770381ms [2022-03-21 06:32:52,896] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where title is not null [2022-03-21 06:32:52,899] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+title+is+not+null [2022-03-21 06:32:52,903] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 2.541469ms [2022-03-21 06:32:52,906] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where firstName is not null [2022-03-21 06:32:52,908] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+firstName+is+not+null [2022-03-21 06:32:52,911] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 1.593684ms [2022-03-21 06:32:52,914] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where lastName is not null [2022-03-21 06:32:52,917] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+lastName+is+not+null [2022-03-21 06:32:52,921] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 2.494285ms [2022-03-21 06:32:52,924] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where streetAddress is not null [2022-03-21 06:32:52,927] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+streetAddress+is+not+null [2022-03-21 06:32:52,931] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 2.234711ms [2022-03-21 06:32:52,933] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where suffix is not null [2022-03-21 06:32:52,936] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+suffix+is+not+null [2022-03-21 06:32:52,940] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 2.329638ms [2022-03-21 06:32:52,943] - [tuq_helper] [139784139831040] - INFO - RUN QUERY select count(*) from default:test_bucket.test_scope_1.test_collection_1 where filler1 is not null [2022-03-21 06:32:52,945] - [rest_client] [139784139831040] - INFO - query params : statement=select+count%28%2A%29+from+default%3Atest_bucket.test_scope_1.test_collection_1+where+filler1+is+not+null [2022-03-21 06:32:52,948] - [tuq_helper] [139784139831040] - INFO - TOTAL ELAPSED TIME: 1.551377ms [2022-03-21 06:32:53,468] - [rest_client] [139784097867520] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235,ns_1@172.23.97.145', 'ejectedNodes': 'ns_1@172.23.107.235', 'user': 'Administrator', 'password': 'password'} [2022-03-21 06:32:53,520] - [rest_client] [139784097867520] - INFO - rebalance operation started [2022-03-21 06:33:03,533] - [rest_client] [139784097867520] - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2022-03-21 06:33:03,537] - [rest_client] [139784097867520] - INFO - Latest logs from UI on 172.23.97.145: [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1647869578530, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {worker_died,\n {\'EXIT\',<0.2546.3>,\n {{badmatch,\n {error,\n {bad_nodes,index,prepare_rebalance,\n [{\'ns_1@172.23.107.235\',\n {error,\n {unknown_error,\n <<"indexer rebalance failure - cleanup pending from previous failed/aborted rebalance/failover/move index. please retry the request later.">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 7f9e9cde4bd62a5ce5f0136b669afe44', 'serverTime': '2022-03-21T06:32:58.530Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.107.235', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1647869577227, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.235\'. Version: "7.1.0-2506-enterprise".', 'serverTime': '2022-03-21T06:32:57.227Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.107.235', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1647869577127, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.107.235' saw that node 'ns_1@172.23.97.145' came up. Tags: []", 'serverTime': '2022-03-21T06:32:57.127Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1647869577124, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.97.145' saw that node 'ns_1@172.23.107.235' came up. Tags: []", 'serverTime': '2022-03-21T06:32:57.124Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.107.235', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1647869577117, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.107.235' saw that node 'ns_1@172.23.106.184' came up. Tags: []", 'serverTime': '2022-03-21T06:32:57.117Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.106.184', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1647869577116, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.106.184' saw that node 'ns_1@172.23.107.235' came up. Tags: []", 'serverTime': '2022-03-21T06:32:57.116Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1647869573470, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.106.184','ns_1@172.23.97.145'], EjectNodes = ['ns_1@172.23.107.235'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 7f9e9cde4bd62a5ce5f0136b669afe44", 'serverTime': '2022-03-21T06:32:53.470Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.106.184', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1647869560182, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.106.184' saw that node 'ns_1@172.23.107.235' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-21T06:32:40.182Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1647869560177, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.97.145' saw that node 'ns_1@172.23.107.235' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-21T06:32:40.177Z'} [2022-03-21 06:33:03,538] - [rest_client] [139784097867520] - ERROR - {'node': 'ns_1@172.23.97.145', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1647869559665, 'shortText': 'message', 'text': "Rebalance exited with reason {service_rebalance_failed,index,\n {agent_died,<34218.13277.0>,\n {lost_connection,\n {'ns_1@172.23.107.235',shutdown}}}}.\nRebalance Operation Id = f1f1000d922fe463c3f10862a5cdc580", 'serverTime': '2022-03-21T06:32:39.665Z'} [2022-03-21 06:33:03,539] - [collections_indexes_rebalance] [139784139831040] - INFO - ============== ConcurrentIndexes tearDown has started ============== [2022-03-21 06:33:03,551] - [basetestcase] [139784139831040] - INFO - list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.235 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:33:03,552] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:33:03,722] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:33:04,093] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:04,538] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:04,550] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.235:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-03-21 06:33:04,551] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.235: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l [2022-03-21 06:33:04,647] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:33:04,649] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:33:04,753] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:33:05,024] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:05,327] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:05,333] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.97.145:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-03-21 06:33:05,334] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l [2022-03-21 06:33:05,398] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:33:05,408] - [basetestcase] [139784139831040] - INFO - list of kv nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:33:05,409] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:33:05,509] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:33:05,783] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:06,088] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:06,094] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.97.145:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-03-21 06:33:06,094] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l [2022-03-21 06:33:06,384] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:33:06,393] - [basetestcase] [139784139831040] - INFO - list of n1ql nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:33:06,393] - [tuq_helper] [139784139831040] - INFO - CHECK FOR PRIMARY INDEXES [2022-03-21 06:33:06,400] - [basetestcase] [139784139831040] - INFO - ------- Cluster statistics ------- [2022-03-21 06:33:06,400] - [basetestcase] [139784139831040] - INFO - 172.23.106.184:8091 => {'services': ['index'], 'cpu_utilization': 4.111675126903553, 'mem_free': 2693926912, 'mem_total': 3945775104, 'swap_mem_used': 44564480, 'swap_mem_total': 3758092288} [2022-03-21 06:33:06,400] - [basetestcase] [139784139831040] - INFO - 172.23.107.235:8091 => {'services': ['index'], 'cpu_utilization': 6.610515569167942, 'mem_free': 2768777216, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:33:06,400] - [basetestcase] [139784139831040] - INFO - 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 2.02668034889687, 'mem_free': 2454786048, 'mem_total': 3935289344, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:33:06,401] - [basetestcase] [139784139831040] - INFO - --- End of cluster statistics --- [2022-03-21 06:33:06,403] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:33:06,500] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:33:06,763] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:07,067] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:07,068] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:33:07,239] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:33:07,613] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:08,046] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:08,049] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:33:08,216] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:33:08,571] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:09,004] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:09,006] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:33:09,181] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:33:09,565] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:10,005] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:10,007] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:33:10,179] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:33:10,576] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:11,019] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:11,021] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:33:11,194] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:33:11,591] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:12,018] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:25,066] - [basetestcase] [139784139831040] - INFO - ============== basetestcase cleanup was started for test #2 test_rebalance_out_of_nodes_with_failed_rebalance ============== [2022-03-21 06:33:25,090] - [bucket_helper] [139784139831040] - INFO - deleting existing buckets ['test_bucket'] on 172.23.97.145 [2022-03-21 06:33:26,089] - [bucket_helper] [139784139831040] - INFO - waiting for bucket deletion to complete.... [2022-03-21 06:33:26,091] - [rest_client] [139784139831040] - INFO - node 172.23.97.145 existing buckets : [] [2022-03-21 06:33:26,091] - [bucket_helper] [139784139831040] - INFO - deleted bucket : test_bucket from 172.23.97.145 [2022-03-21 06:33:26,103] - [bucket_helper] [139784139831040] - INFO - Could not find any buckets for node 172.23.106.184, nothing to delete [2022-03-21 06:33:26,116] - [bucket_helper] [139784139831040] - INFO - Could not find any buckets for node 172.23.107.235, nothing to delete [2022-03-21 06:33:26,127] - [bucket_helper] [139784139831040] - INFO - Could not find any buckets for node 172.23.107.230, nothing to delete [2022-03-21 06:33:26,140] - [bucket_helper] [139784139831040] - INFO - Could not find any buckets for node 172.23.107.24, nothing to delete [2022-03-21 06:33:26,149] - [bucket_helper] [139784139831040] - INFO - Could not find any buckets for node 172.23.107.247, nothing to delete [2022-03-21 06:33:26,151] - [basetestcase] [139784139831040] - INFO - Removing user 'clientuser'... [2022-03-21 06:33:26,160] - [rest_client] [139784139831040] - ERROR - DELETE http://172.23.97.145:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-21 06:33:26,160] - [basetestcase] [139784139831040] - INFO - b'"User was not found."' [2022-03-21 06:33:26,163] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:26,174] - [cluster_helper] [139784139831040] - INFO - rebalancing all nodes in order to remove nodes [2022-03-21 06:33:26,177] - [rest_client] [139784139831040] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235,ns_1@172.23.97.145', 'ejectedNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235', 'user': 'Administrator', 'password': 'password'} [2022-03-21 06:33:26,235] - [rest_client] [139784139831040] - INFO - rebalance operation started [2022-03-21 06:33:26,237] - [rest_client] [139784139831040] - INFO - rebalance percentage : 0.00 % [2022-03-21 06:33:46,263] - [rest_client] [139784139831040] - INFO - rebalance progress took 20.03 seconds [2022-03-21 06:33:46,263] - [rest_client] [139784139831040] - INFO - sleep for 10 seconds after rebalance... [2022-03-21 06:33:56,296] - [cluster_helper] [139784139831040] - INFO - removed all the nodes from cluster associated with ip:172.23.97.145 port:8091 ssh_username:root ? [('ns_1@172.23.106.184', 8091), ('ns_1@172.23.107.235', 8091)] [2022-03-21 06:33:56,300] - [cluster_helper] [139784139831040] - INFO - waiting for ns_server @ 172.23.97.145:8091 [2022-03-21 06:33:56,300] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:56,303] - [cluster_helper] [139784139831040] - INFO - ns_server @ 172.23.97.145:8091 is running [2022-03-21 06:33:56,308] - [cluster_helper] [139784139831040] - INFO - waiting for ns_server @ 172.23.106.184:8091 [2022-03-21 06:33:56,309] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:56,314] - [cluster_helper] [139784139831040] - INFO - ns_server @ 172.23.106.184:8091 is running [2022-03-21 06:33:56,321] - [cluster_helper] [139784139831040] - INFO - waiting for ns_server @ 172.23.107.235:8091 [2022-03-21 06:33:56,321] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:56,326] - [cluster_helper] [139784139831040] - INFO - ns_server @ 172.23.107.235:8091 is running [2022-03-21 06:33:56,333] - [cluster_helper] [139784139831040] - INFO - waiting for ns_server @ 172.23.107.230:8091 [2022-03-21 06:33:56,333] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:56,338] - [cluster_helper] [139784139831040] - INFO - ns_server @ 172.23.107.230:8091 is running [2022-03-21 06:33:56,345] - [cluster_helper] [139784139831040] - INFO - waiting for ns_server @ 172.23.107.24:8091 [2022-03-21 06:33:56,345] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:56,350] - [cluster_helper] [139784139831040] - INFO - ns_server @ 172.23.107.24:8091 is running [2022-03-21 06:33:56,357] - [cluster_helper] [139784139831040] - INFO - waiting for ns_server @ 172.23.107.247:8091 [2022-03-21 06:33:56,358] - [rest_client] [139784139831040] - INFO - -->is_ns_server_running? [2022-03-21 06:33:56,363] - [cluster_helper] [139784139831040] - INFO - ns_server @ 172.23.107.247:8091 is running [2022-03-21 06:33:56,363] - [basetestcase] [139784139831040] - INFO - ============== basetestcase cleanup was finished for test #2 test_rebalance_out_of_nodes_with_failed_rebalance ============== [2022-03-21 06:33:56,363] - [basetestcase] [139784139831040] - INFO - closing all ssh connections [2022-03-21 06:33:56,364] - [basetestcase] [139784139831040] - INFO - closing all memcached connections [2022-03-21 06:33:56,369] - [collections_indexes_rebalance] [139784139831040] - INFO - ============== ConcurrentIndexes tearDown has completed ============== [2022-03-21 06:33:56,372] - [remote_util] [139782933165824] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:33:56,373] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:33:56,374] - [remote_util] [139782949951232] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:33:56,376] - [remote_util] [139782966736640] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:33:56,378] - [remote_util] [139782958343936] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:33:56,379] - [remote_util] [139782924773120] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:33:56,476] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:33:56,539] - [remote_util] [139782933165824] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:33:56,540] - [remote_util] [139782958343936] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:33:56,544] - [remote_util] [139782966736640] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:33:56,547] - [remote_util] [139782924773120] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:33:56,551] - [remote_util] [139782949951232] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:33:56,743] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:56,926] - [remote_util] [139782949951232] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:56,939] - [remote_util] [139782924773120] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:56,941] - [remote_util] [139782933165824] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:56,943] - [remote_util] [139782958343936] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:56,947] - [remote_util] [139782966736640] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:33:57,042] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:57,043] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.97.145: /opt/couchbase/bin/cbcollect_info 172.23.97.145-20220321-0633-diag.zip [2022-03-21 06:33:57,375] - [remote_util] [139782949951232] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:57,375] - [remote_util] [139782949951232] - INFO - running command.raw on 172.23.107.235: /opt/couchbase/bin/cbcollect_info 172.23.107.235-20220321-0633-diag.zip [2022-03-21 06:33:57,379] - [remote_util] [139782958343936] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:57,379] - [remote_util] [139782958343936] - INFO - running command.raw on 172.23.107.230: /opt/couchbase/bin/cbcollect_info 172.23.107.230-20220321-0633-diag.zip [2022-03-21 06:33:57,380] - [remote_util] [139782924773120] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:57,380] - [remote_util] [139782924773120] - INFO - running command.raw on 172.23.107.247: /opt/couchbase/bin/cbcollect_info 172.23.107.247-20220321-0633-diag.zip [2022-03-21 06:33:57,384] - [remote_util] [139782933165824] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:57,384] - [remote_util] [139782933165824] - INFO - running command.raw on 172.23.106.184: /opt/couchbase/bin/cbcollect_info 172.23.106.184-20220321-0633-diag.zip [2022-03-21 06:33:57,390] - [remote_util] [139782966736640] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:33:57,391] - [remote_util] [139782966736640] - INFO - running command.raw on 172.23.107.24: /opt/couchbase/bin/cbcollect_info 172.23.107.24-20220321-0633-diag.zip [2022-03-21 06:35:55,136] - [remote_util] [139782966736640] - INFO - command executed successfully with root [2022-03-21 06:35:55,225] - [remote_util] [139782958343936] - INFO - command executed successfully with root [2022-03-21 06:35:55,390] - [remote_util] [139782966736640] - INFO - running command.raw on 172.23.107.24: rm -f /root/172.23.107.24-20220321-0633-diag.zip [2022-03-21 06:35:55,424] - [remote_util] [139782966736640] - INFO - command executed successfully with root [2022-03-21 06:35:55,485] - [remote_util] [139782958343936] - INFO - running command.raw on 172.23.107.230: rm -f /root/172.23.107.230-20220321-0633-diag.zip [2022-03-21 06:35:55,522] - [remote_util] [139782958343936] - INFO - command executed successfully with root [2022-03-21 06:35:55,991] - [remote_util] [139782924773120] - INFO - command executed successfully with root [2022-03-21 06:35:56,238] - [remote_util] [139782924773120] - INFO - running command.raw on 172.23.107.247: rm -f /root/172.23.107.247-20220321-0633-diag.zip [2022-03-21 06:35:56,272] - [remote_util] [139782924773120] - INFO - command executed successfully with root [2022-03-21 06:35:58,566] - [remote_util] [139782933165824] - INFO - command executed successfully with root [2022-03-21 06:35:58,840] - [remote_util] [139782933165824] - INFO - running command.raw on 172.23.106.184: rm -f /root/172.23.106.184-20220321-0633-diag.zip [2022-03-21 06:35:58,877] - [remote_util] [139782933165824] - INFO - command executed successfully with root [2022-03-21 06:36:15,484] - [remote_util] [139782949951232] - INFO - command executed successfully with root [2022-03-21 06:36:15,749] - [remote_util] [139782949951232] - INFO - running command.raw on 172.23.107.235: rm -f /root/172.23.107.235-20220321-0633-diag.zip [2022-03-21 06:36:15,783] - [remote_util] [139782949951232] - INFO - command executed successfully with root [2022-03-21 06:36:53,358] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:36:53,709] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.97.145: rm -f /root/172.23.97.145-20220321-0633-diag.zip [2022-03-21 06:36:53,727] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:36:53,990] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:36:54,095] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:36:54,367] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:36:54,673] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:36:54,678] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:36:54,688] - [rest_client] [139785413363520] - INFO - Node version in cluster 7.1.0-2506-enterprise [2022-03-21 06:36:54,688] - [basetestcase] [139785413363520] - INFO - ============== basetestcase setup was started for test #2 suite_tearDown============== [2022-03-21 06:36:54,689] - [basetestcase] [139785413363520] - INFO - Building docker image with java sdk client [2022-03-21 06:36:58,261] - [basetestcase] [139785413363520] - INFO - initializing cluster [2022-03-21 06:36:59,000] - [task] [139784139831040] - INFO - server: ip:172.23.97.145 port:8091 ssh_username:root, nodes/self [2022-03-21 06:36:59,004] - [task] [139784139831040] - INFO - {'uptime': '1852', 'memoryTotal': 3935289344, 'memoryFree': 2704887808, 'mcdMemoryReserved': 3002, 'mcdMemoryAllocated': 3002, 'status': 'healthy', 'hostname': '172.23.97.145:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1006, 'memcached': 11210, 'id': 'ns_1@172.23.97.145', 'ip': '172.23.97.145', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 3752, 'curr_items': 0} [2022-03-21 06:36:59,004] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1005 [2022-03-21 06:36:59,014] - [task] [139784139831040] - INFO - quota for index service will be 1005 MB [2022-03-21 06:36:59,014] - [task] [139784139831040] - INFO - set index quota to node 172.23.97.145 [2022-03-21 06:36:59,014] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1005 [2022-03-21 06:36:59,019] - [rest_client] [139784139831040] - INFO - pools/default params : memoryQuota=1006 [2022-03-21 06:36:59,024] - [rest_client] [139784139831040] - INFO - --> init_node_services(Administrator,password,172.23.97.145,8091,['kv', 'n1ql', 'index']) [2022-03-21 06:36:59,024] - [rest_client] [139784139831040] - INFO - /node/controller/setupServices params on 172.23.97.145: 8091:hostname=172.23.97.145&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex [2022-03-21 06:36:59,026] - [rest_client] [139784139831040] - ERROR - POST http://172.23.97.145:8091//node/controller/setupServices body: hostname=172.23.97.145&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-21 06:36:59,026] - [rest_client] [139784139831040] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-21 06:36:59,026] - [rest_client] [139784139831040] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:36:59,026] - [rest_client] [139784139831040] - INFO - settings/web params on 172.23.97.145:8091:port=8091&username=Administrator&password=password [2022-03-21 06:36:59,077] - [rest_client] [139784139831040] - INFO - --> status:True [2022-03-21 06:36:59,078] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:36:59,191] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:36:59,464] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:36:59,765] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:36:59,766] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:36:59,831] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:36:59,832] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:36:59,834] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.97.145:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:36:59,836] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.97.145:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:36:59,844] - [rest_client] [139784139831040] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:36:59,862] - [task] [139784139831040] - INFO - server: ip:172.23.106.184 port:8091 ssh_username:root, nodes/self [2022-03-21 06:36:59,869] - [task] [139784139831040] - INFO - {'uptime': '1842', 'memoryTotal': 3945775104, 'memoryFree': 3279118336, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.106.184:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.106.184', 'ip': '172.23.106.184', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:36:59,869] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:36:59,879] - [rest_client] [139784139831040] - ERROR - GET http://172.23.106.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-21 06:36:59,880] - [rest_client] [139784139831040] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:36:59,886] - [rest_client] [139784139831040] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:36:59,886] - [rest_client] [139784139831040] - INFO - settings/web params on 172.23.106.184:8091:port=8091&username=Administrator&password=password [2022-03-21 06:36:59,967] - [rest_client] [139784139831040] - INFO - --> status:True [2022-03-21 06:36:59,968] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:37:00,134] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:37:00,509] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:37:00,943] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:37:00,944] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.106.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:37:01,004] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:37:01,004] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:37:01,009] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.106.184:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:01,012] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.106.184:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:01,019] - [rest_client] [139784139831040] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:37:01,040] - [task] [139784139831040] - INFO - server: ip:172.23.107.235 port:8091 ssh_username:root, nodes/self [2022-03-21 06:37:01,046] - [task] [139784139831040] - INFO - {'uptime': '248', 'memoryTotal': 3945775104, 'memoryFree': 3271917568, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.235:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'memcached': 11210, 'id': 'ns_1@172.23.107.235', 'ip': '172.23.107.235', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:37:01,046] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:37:01,057] - [rest_client] [139784139831040] - ERROR - GET http://172.23.107.235:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-21 06:37:01,057] - [rest_client] [139784139831040] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:37:01,064] - [rest_client] [139784139831040] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:37:01,064] - [rest_client] [139784139831040] - INFO - settings/web params on 172.23.107.235:8091:port=8091&username=Administrator&password=password [2022-03-21 06:37:01,130] - [rest_client] [139784139831040] - INFO - --> status:True [2022-03-21 06:37:01,132] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:37:01,267] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:37:01,649] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:37:02,086] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:37:02,087] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.235: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:37:02,140] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:37:02,140] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:37:02,144] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.235:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:02,147] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.235:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:02,156] - [rest_client] [139784139831040] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:37:02,184] - [task] [139784139831040] - INFO - server: ip:172.23.107.230 port:8091 ssh_username:root, nodes/self [2022-03-21 06:37:02,191] - [task] [139784139831040] - INFO - {'uptime': '1840', 'memoryTotal': 3945775104, 'memoryFree': 3257180160, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.230:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2016, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.230', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:37:02,191] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:37:02,205] - [rest_client] [139784139831040] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:37:02,212] - [rest_client] [139784139831040] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:37:02,212] - [rest_client] [139784139831040] - INFO - settings/web params on 172.23.107.230:8091:port=8091&username=Administrator&password=password [2022-03-21 06:37:02,293] - [rest_client] [139784139831040] - INFO - --> status:True [2022-03-21 06:37:02,294] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:37:02,479] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:37:02,883] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:37:03,322] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:37:03,323] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.230: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:37:03,376] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:37:03,376] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:37:03,380] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.230:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:03,383] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.230:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:03,390] - [rest_client] [139784139831040] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:37:03,411] - [task] [139784139831040] - INFO - server: ip:172.23.107.24 port:8091 ssh_username:root, nodes/self [2022-03-21 06:37:03,416] - [task] [139784139831040] - INFO - {'uptime': '1845', 'memoryTotal': 3945775104, 'memoryFree': 3261181952, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.24:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2016, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.24', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:37:03,416] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:37:03,432] - [rest_client] [139784139831040] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:37:03,439] - [rest_client] [139784139831040] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:37:03,439] - [rest_client] [139784139831040] - INFO - settings/web params on 172.23.107.24:8091:port=8091&username=Administrator&password=password [2022-03-21 06:37:03,496] - [rest_client] [139784139831040] - INFO - --> status:True [2022-03-21 06:37:03,497] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:37:03,667] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:37:04,037] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:37:04,476] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:37:04,476] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.24: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:37:04,570] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:37:04,571] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:37:04,575] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.24:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:04,578] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.24:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:04,586] - [rest_client] [139784139831040] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:37:04,610] - [task] [139784139831040] - INFO - server: ip:172.23.107.247 port:8091 ssh_username:root, nodes/self [2022-03-21 06:37:04,616] - [task] [139784139831040] - INFO - {'uptime': '1845', 'memoryTotal': 3945775104, 'memoryFree': 3254943744, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.247:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2506-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2016, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.247', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2022-03-21 06:37:04,616] - [rest_client] [139784139831040] - INFO - pools/default params : indexMemoryQuota=1008 [2022-03-21 06:37:04,633] - [rest_client] [139784139831040] - INFO - pools/default params : memoryQuota=2016 [2022-03-21 06:37:04,638] - [rest_client] [139784139831040] - INFO - --> in init_cluster...Administrator,password,8091 [2022-03-21 06:37:04,639] - [rest_client] [139784139831040] - INFO - settings/web params on 172.23.107.247:8091:port=8091&username=Administrator&password=password [2022-03-21 06:37:04,696] - [rest_client] [139784139831040] - INFO - --> status:True [2022-03-21 06:37:04,698] - [remote_util] [139784139831040] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:37:04,851] - [remote_util] [139784139831040] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:37:05,240] - [remote_util] [139784139831040] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:37:05,671] - [remote_util] [139784139831040] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:37:05,672] - [remote_util] [139784139831040] - INFO - running command.raw on 172.23.107.247: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:37:05,763] - [remote_util] [139784139831040] - INFO - command executed successfully with root [2022-03-21 06:37:05,763] - [remote_util] [139784139831040] - INFO - ['ok'] [2022-03-21 06:37:05,768] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.247:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:05,772] - [rest_client] [139784139831040] - INFO - /diag/eval status on 172.23.107.247:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-21 06:37:05,779] - [rest_client] [139784139831040] - INFO - settings/indexes params : storageMode=plasma [2022-03-21 06:37:05,794] - [basetestcase] [139785413363520] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.97.145 **** [2022-03-21 06:37:05,878] - [basetestcase] [139785413363520] - INFO - sleep for 5 secs. ... [2022-03-21 06:37:10,884] - [basetestcase] [139785413363520] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-21 06:37:10,895] - [basetestcase] [139785413363520] - INFO - done initializing cluster [2022-03-21 06:37:11,800] - [task] [139784139831040] - INFO - adding node 172.23.106.184:8091 to cluster [2022-03-21 06:37:11,807] - [rest_client] [139784139831040] - INFO - adding remote node @172.23.106.184:18091 to this cluster @172.23.97.145:8091 [2022-03-21 06:37:22,879] - [rest_client] [139784139831040] - INFO - rebalance progress took 11.07 seconds [2022-03-21 06:37:22,879] - [rest_client] [139784139831040] - INFO - sleep for 10 seconds after rebalance... [2022-03-21 06:37:37,108] - [task] [139784139831040] - INFO - adding node 172.23.107.235:8091 to cluster [2022-03-21 06:37:37,116] - [rest_client] [139784139831040] - INFO - adding remote node @172.23.107.235:18091 to this cluster @172.23.97.145:8091 [2022-03-21 06:37:47,130] - [rest_client] [139784139831040] - INFO - rebalance progress took 10.01 seconds [2022-03-21 06:37:47,131] - [rest_client] [139784139831040] - INFO - sleep for 10 seconds after rebalance... [2022-03-21 06:38:01,127] - [rest_client] [139784139831040] - INFO - Node 172.23.106.184 not part of cluster inactiveAdded [2022-03-21 06:38:01,127] - [rest_client] [139784139831040] - INFO - Node 172.23.107.235 not part of cluster inactiveAdded [2022-03-21 06:38:01,137] - [rest_client] [139784139831040] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235,ns_1@172.23.97.145', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-21 06:38:01,191] - [rest_client] [139784139831040] - INFO - rebalance operation started [2022-03-21 06:38:11,204] - [task] [139784139831040] - INFO - Rebalance - status: none, progress: 100.00% [2022-03-21 06:38:11,208] - [task] [139784139831040] - INFO - rebalancing was completed with progress: 100% in 10.017592430114746 sec [2022-03-21 06:38:11,211] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:38:11,315] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:38:11,587] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:11,889] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:11,890] - [remote_util] [139785413363520] - INFO - running command.raw on 172.23.97.145: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-21 06:38:11,953] - [remote_util] [139785413363520] - INFO - command executed successfully with root [2022-03-21 06:38:11,953] - [remote_util] [139785413363520] - INFO - ['ok'] [2022-03-21 06:38:11,953] - [basetestcase] [139785413363520] - INFO - Enabled diag/eval for non-local hosts from 172.23.97.145 [2022-03-21 06:38:11,953] - [basetestcase] [139785413363520] - INFO - ============== basetestcase setup was finished for test #2 suite_tearDown ============== [2022-03-21 06:38:11,960] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:38:12,063] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:38:12,327] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:12,630] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:12,632] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:38:12,803] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:38:13,188] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:13,630] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:13,632] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:38:13,803] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:38:14,167] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:14,604] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:14,606] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:38:14,776] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:38:15,174] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:15,623] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:15,625] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:38:15,789] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:38:16,192] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:16,631] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:16,633] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:38:16,799] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:38:17,174] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:17,604] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:30,302] - [basetestcase] [139785413363520] - INFO - ------- Cluster statistics ------- [2022-03-21 06:38:30,302] - [basetestcase] [139785413363520] - INFO - 172.23.106.184:8091 => {'services': ['index'], 'cpu_utilization': 11.22629582806574, 'mem_free': 3246149632, 'mem_total': 3945775104, 'swap_mem_used': 44564480, 'swap_mem_total': 3758092288} [2022-03-21 06:38:30,302] - [basetestcase] [139785413363520] - INFO - 172.23.107.235:8091 => {'services': ['index'], 'cpu_utilization': 3.367941250949607, 'mem_free': 3246825472, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:38:30,302] - [basetestcase] [139785413363520] - INFO - 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 11.17855336368235, 'mem_free': 2713649152, 'mem_total': 3935289344, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:38:30,302] - [basetestcase] [139785413363520] - INFO - --- End of cluster statistics --- [2022-03-21 06:38:30,302] - [basetestcase] [139785413363520] - INFO - Time to execute basesetup : 96.31340980529785 [2022-03-21 06:38:30,309] - [newtuq] [139785413363520] - INFO - Initial status of 172.23.97.145 cluster is healthy [2022-03-21 06:38:30,309] - [newtuq] [139785413363520] - INFO - current status of 172.23.97.145 is healthy [2022-03-21 06:38:30,329] - [newtuq] [139785413363520] - INFO - Initial status of 172.23.106.184 cluster is healthy [2022-03-21 06:38:30,329] - [newtuq] [139785413363520] - INFO - current status of 172.23.106.184 is healthy [2022-03-21 06:38:30,350] - [newtuq] [139785413363520] - INFO - Initial status of 172.23.107.235 cluster is healthy [2022-03-21 06:38:30,350] - [newtuq] [139785413363520] - INFO - current status of 172.23.107.235 is healthy [2022-03-21 06:38:30,364] - [newtuq] [139785413363520] - INFO - Initial status of 172.23.107.230 cluster is healthy [2022-03-21 06:38:30,364] - [newtuq] [139785413363520] - INFO - current status of 172.23.107.230 is healthy [2022-03-21 06:38:30,377] - [newtuq] [139785413363520] - INFO - Initial status of 172.23.107.24 cluster is healthy [2022-03-21 06:38:30,377] - [newtuq] [139785413363520] - INFO - current status of 172.23.107.24 is healthy [2022-03-21 06:38:30,390] - [newtuq] [139785413363520] - INFO - Initial status of 172.23.107.247 cluster is healthy [2022-03-21 06:38:30,390] - [newtuq] [139785413363520] - INFO - current status of 172.23.107.247 is healthy [2022-03-21 06:38:30,396] - [basetestcase] [139785413363520] - INFO - list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.235 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:38:30,420] - [rest_client] [139785413363520] - INFO - {'indexer.settings.storage_mode': 'plasma'} set [2022-03-21 06:38:30,420] - [newtuq] [139785413363520] - INFO - Allowing the indexer to complete restart after setting the internal settings [2022-03-21 06:38:30,420] - [basetestcase] [139785413363520] - INFO - sleep for 5 secs. ... [2022-03-21 06:38:35,441] - [rest_client] [139785413363520] - INFO - {'indexer.api.enableTestServer': True} set [2022-03-21 06:38:35,442] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:38:35,542] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:38:35,813] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:38:36,117] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:38:43,063] - [basetestcase] [139785413363520] - INFO - LOAD IS FINISHED [2022-03-21 06:38:43,073] - [basetestcase] [139785413363520] - INFO - list of n1ql nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:38:43,073] - [newtuq] [139785413363520] - INFO - ip:172.23.97.145 port:8091 ssh_username:root [2022-03-21 06:38:43,074] - [basetestcase] [139785413363520] - INFO - sleep for 30 secs. ... [2022-03-21 06:39:13,101] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:39:13,203] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:39:13,467] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:13,771] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:13,786] - [basetestcase] [139785413363520] - INFO - list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.235 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:39:13,795] - [rest_client] [139785413363520] - INFO - Making a rest request api=http://172.23.106.184:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False [2022-03-21 06:39:13,810] - [rest_client] [139785413363520] - INFO - {'queryport.client.waitForScheduledIndex': False} set [2022-03-21 06:39:13,811] - [rest_client] [139785413363520] - INFO - Making a rest request api=http://172.23.106.184:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False [2022-03-21 06:39:13,831] - [rest_client] [139785413363520] - INFO - {'indexer.allowScheduleCreateRebal': True} set [2022-03-21 06:39:13,831] - [collections_indexes_rebalance] [139785413363520] - INFO - ============== ConcurrentIndexes setup has started ============== [2022-03-21 06:39:14,503] - [rest_client] [139784139831040] - INFO - http://172.23.97.145:8091/pools/default/buckets with param: name=test_bucket&ramQuotaMB=256&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-21 06:39:14,534] - [rest_client] [139784139831040] - INFO - 0.03 seconds to create bucket test_bucket [2022-03-21 06:39:14,534] - [bucket_helper] [139784139831040] - INFO - waiting for memcached bucket : test_bucket in 172.23.97.145 to accept set ops [2022-03-21 06:39:15,110] - [data_helper] [139784139831040] - INFO - creating direct client 172.23.97.145:11210 test_bucket [2022-03-21 06:39:15,169] - [data_helper] [139784139831040] - INFO - creating direct client 172.23.97.145:11210 test_bucket [2022-03-21 06:39:16,555] - [task] [139784139831040] - INFO - bucket 'test_bucket' was created with per node RAM quota: 256 [2022-03-21 06:39:16,570] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:39:16,672] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:39:16,940] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:17,243] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:17,243] - [collections_indexes_rebalance] [139785413363520] - INFO - ============== ConcurrentIndexes setup has completed ============== [2022-03-21 06:39:17,244] - [collections_indexes_rebalance] [139785413363520] - INFO - ============== ConcurrentIndexes tearDown has started ============== [2022-03-21 06:39:17,256] - [basetestcase] [139785413363520] - INFO - list of index nodes in cluster: [ip:172.23.106.184 port:8091 ssh_username:root, ip:172.23.107.235 port:8091 ssh_username:root, ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:39:17,257] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:39:17,424] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:39:17,789] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:18,233] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:18,248] - [rest_client] [139785413363520] - INFO - /diag/eval status on 172.23.107.235:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-03-21 06:39:18,248] - [remote_util] [139785413363520] - INFO - running command.raw on 172.23.107.235: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l [2022-03-21 06:39:18,338] - [remote_util] [139785413363520] - INFO - command executed successfully with root [2022-03-21 06:39:18,340] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:39:18,441] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:39:18,700] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:19,002] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:19,009] - [rest_client] [139785413363520] - INFO - /diag/eval status on 172.23.97.145:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-03-21 06:39:19,010] - [remote_util] [139785413363520] - INFO - running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/indexer.log* | wc -l [2022-03-21 06:39:19,073] - [remote_util] [139785413363520] - INFO - command executed successfully with root [2022-03-21 06:39:19,083] - [basetestcase] [139785413363520] - INFO - list of kv nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:39:19,084] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:39:19,180] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:39:19,447] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:19,754] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:19,761] - [rest_client] [139785413363520] - INFO - /diag/eval status on 172.23.97.145:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2022-03-21 06:39:19,761] - [remote_util] [139785413363520] - INFO - running command.raw on 172.23.97.145: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/projector.log* | wc -l [2022-03-21 06:39:20,044] - [remote_util] [139785413363520] - INFO - command executed successfully with root [2022-03-21 06:39:20,050] - [basetestcase] [139785413363520] - INFO - list of n1ql nodes in cluster: [ip:172.23.97.145 port:8091 ssh_username:root] [2022-03-21 06:39:20,051] - [tuq_helper] [139785413363520] - INFO - CHECK FOR PRIMARY INDEXES [2022-03-21 06:39:20,056] - [basetestcase] [139785413363520] - INFO - ------- Cluster statistics ------- [2022-03-21 06:39:20,056] - [basetestcase] [139785413363520] - INFO - 172.23.106.184:8091 => {'services': ['index'], 'cpu_utilization': 3.222557905337362, 'mem_free': 3229171712, 'mem_total': 3945775104, 'swap_mem_used': 44564480, 'swap_mem_total': 3758092288} [2022-03-21 06:39:20,056] - [basetestcase] [139785413363520] - INFO - 172.23.107.235:8091 => {'services': ['index'], 'cpu_utilization': 2.451971688574317, 'mem_free': 3237986304, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:39:20,056] - [basetestcase] [139785413363520] - INFO - 172.23.97.145:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.786612984398591, 'mem_free': 2736078848, 'mem_total': 3935289344, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2022-03-21 06:39:20,056] - [basetestcase] [139785413363520] - INFO - --- End of cluster statistics --- [2022-03-21 06:39:20,057] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.97.145 with username:root, attempt#1 of 5 [2022-03-21 06:39:20,161] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.97.145 as root [2022-03-21 06:39:20,420] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:20,727] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:20,730] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.106.184 with username:root, attempt#1 of 5 [2022-03-21 06:39:20,895] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.106.184 as root [2022-03-21 06:39:21,271] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:21,709] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:21,711] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.235 with username:root, attempt#1 of 5 [2022-03-21 06:39:21,872] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.235 as root [2022-03-21 06:39:22,231] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:22,671] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:22,674] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.230 with username:root, attempt#1 of 5 [2022-03-21 06:39:22,847] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.230 as root [2022-03-21 06:39:23,241] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:23,683] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:23,686] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.24 with username:root, attempt#1 of 5 [2022-03-21 06:39:23,857] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.24 as root [2022-03-21 06:39:24,234] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:24,667] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:24,670] - [remote_util] [139785413363520] - INFO - SSH Connecting to 172.23.107.247 with username:root, attempt#1 of 5 [2022-03-21 06:39:24,839] - [remote_util] [139785413363520] - INFO - SSH Connected to 172.23.107.247 as root [2022-03-21 06:39:25,204] - [remote_util] [139785413363520] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-21 06:39:25,636] - [remote_util] [139785413363520] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-21 06:39:38,623] - [basetestcase] [139785413363520] - INFO - ============== basetestcase cleanup was started for test #2 suite_tearDown ============== [2022-03-21 06:39:38,649] - [bucket_helper] [139785413363520] - INFO - deleting existing buckets ['test_bucket'] on 172.23.97.145 [2022-03-21 06:39:39,062] - [bucket_helper] [139785413363520] - INFO - waiting for bucket deletion to complete.... [2022-03-21 06:39:39,065] - [rest_client] [139785413363520] - INFO - node 172.23.97.145 existing buckets : [] [2022-03-21 06:39:39,065] - [bucket_helper] [139785413363520] - INFO - deleted bucket : test_bucket from 172.23.97.145 [2022-03-21 06:39:39,077] - [bucket_helper] [139785413363520] - INFO - Could not find any buckets for node 172.23.106.184, nothing to delete [2022-03-21 06:39:39,091] - [bucket_helper] [139785413363520] - INFO - Could not find any buckets for node 172.23.107.235, nothing to delete [2022-03-21 06:39:39,101] - [bucket_helper] [139785413363520] - INFO - Could not find any buckets for node 172.23.107.230, nothing to delete [2022-03-21 06:39:39,111] - [bucket_helper] [139785413363520] - INFO - Could not find any buckets for node 172.23.107.24, nothing to delete [2022-03-21 06:39:39,120] - [bucket_helper] [139785413363520] - INFO - Could not find any buckets for node 172.23.107.247, nothing to delete [2022-03-21 06:39:39,120] - [basetestcase] [139785413363520] - INFO - Removing user 'clientuser'... [2022-03-21 06:39:39,130] - [rest_client] [139785413363520] - ERROR - DELETE http://172.23.97.145:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-21 06:39:39,130] - [basetestcase] [139785413363520] - INFO - b'"User was not found."' [2022-03-21 06:39:39,133] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:39:39,144] - [cluster_helper] [139785413363520] - INFO - rebalancing all nodes in order to remove nodes [2022-03-21 06:39:39,146] - [rest_client] [139785413363520] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235,ns_1@172.23.97.145', 'ejectedNodes': 'ns_1@172.23.106.184,ns_1@172.23.107.235', 'user': 'Administrator', 'password': 'password'} [2022-03-21 06:39:39,198] - [rest_client] [139785413363520] - INFO - rebalance operation started [2022-03-21 06:39:39,200] - [rest_client] [139785413363520] - INFO - rebalance percentage : 0.00 % [2022-03-21 06:39:59,226] - [rest_client] [139785413363520] - INFO - rebalance progress took 20.03 seconds [2022-03-21 06:39:59,227] - [rest_client] [139785413363520] - INFO - sleep for 10 seconds after rebalance... [2022-03-21 06:40:09,259] - [cluster_helper] [139785413363520] - INFO - removed all the nodes from cluster associated with ip:172.23.97.145 port:8091 ssh_username:root ? [('ns_1@172.23.106.184', 8091), ('ns_1@172.23.107.235', 8091)] [2022-03-21 06:40:09,262] - [cluster_helper] [139785413363520] - INFO - waiting for ns_server @ 172.23.97.145:8091 [2022-03-21 06:40:09,262] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:40:09,265] - [cluster_helper] [139785413363520] - INFO - ns_server @ 172.23.97.145:8091 is running [2022-03-21 06:40:09,271] - [cluster_helper] [139785413363520] - INFO - waiting for ns_server @ 172.23.106.184:8091 [2022-03-21 06:40:09,271] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:40:09,276] - [cluster_helper] [139785413363520] - INFO - ns_server @ 172.23.106.184:8091 is running [2022-03-21 06:40:09,281] - [cluster_helper] [139785413363520] - INFO - waiting for ns_server @ 172.23.107.235:8091 [2022-03-21 06:40:09,281] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:40:09,287] - [cluster_helper] [139785413363520] - INFO - ns_server @ 172.23.107.235:8091 is running [2022-03-21 06:40:09,294] - [cluster_helper] [139785413363520] - INFO - waiting for ns_server @ 172.23.107.230:8091 [2022-03-21 06:40:09,294] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:40:09,300] - [cluster_helper] [139785413363520] - INFO - ns_server @ 172.23.107.230:8091 is running [2022-03-21 06:40:09,306] - [cluster_helper] [139785413363520] - INFO - waiting for ns_server @ 172.23.107.24:8091 [2022-03-21 06:40:09,306] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:40:09,311] - [cluster_helper] [139785413363520] - INFO - ns_server @ 172.23.107.24:8091 is running [2022-03-21 06:40:09,318] - [cluster_helper] [139785413363520] - INFO - waiting for ns_server @ 172.23.107.247:8091 [2022-03-21 06:40:09,318] - [rest_client] [139785413363520] - INFO - -->is_ns_server_running? [2022-03-21 06:40:09,324] - [cluster_helper] [139785413363520] - INFO - ns_server @ 172.23.107.247:8091 is running [2022-03-21 06:40:09,324] - [basetestcase] [139785413363520] - INFO - ============== basetestcase cleanup was finished for test #2 suite_tearDown ============== [2022-03-21 06:40:09,324] - [basetestcase] [139785413363520] - INFO - closing all ssh connections [2022-03-21 06:40:09,325] - [basetestcase] [139785413363520] - INFO - closing all memcached connections [2022-03-21 06:40:09,330] - [collections_indexes_rebalance] [139785413363520] - INFO - ============== ConcurrentIndexes tearDown has completed ==============