test_best_effort_distribution_max_group (fts.fts_server_groups.FTSServerGroups) ... Logs will be stored at /data/workspace/debian-p0-fts-vset00-00-server-groups_7.0_P1/logs/testrunner-23-Sep-20_21-03-01/test_29 ./testrunner -i /data/workspace/debian-p0-fts-vset00-00-server-groups_7.0_P1/testexec.21594.ini -p get-cbcollect-info=True,disable_HTP=True,get-logs=True,stop-on-failure=False,index_type=scorch,fts_quota=3000,skip_log_scan=True,bucket_storage=couchstore,GROUP=P1,sirius_url=http://172.23.120.103:4000 -t fts.fts_server_groups.FTSServerGroups.test_best_effort_distribution_max_group,GROUP=P1,cluster=D,D,D,F,F,F,F,F,F,fts_quota=3000,index_type=scorch,server_groups=sg1-D:F:F|sg2-D:F:F|sg3-D:F:F,replicas=2,partitions=12,eject_nodes=sg3-F:F,eject_type=failover Test Input params: {'GROUP': 'P1', 'cluster': 'D,D,D,F,F,F,F,F,F', 'fts_quota': '3000', 'index_type': 'scorch', 'server_groups': 'sg1-D:F:F|sg2-D:F:F|sg3-D:F:F', 'replicas': '2', 'partitions': '12', 'eject_nodes': 'sg3-F:F', 'eject_type': 'failover', 'ini': '/data/workspace/debian-p0-fts-vset00-00-server-groups_7.0_P1/testexec.21594.ini', 'cluster_name': 'testexec.21594', 'spec': 'py-fts-server-groups', 'conf_file': 'fts/py-fts-server-groups.conf', 'get-cbcollect-info': 'True', 'disable_HTP': 'True', 'get-logs': 'True', 'stop-on-failure': 'False', 'skip_log_scan': 'True', 'bucket_storage': 'couchstore', 'sirius_url': 'http://172.23.120.103:4000', 'num_nodes': 12, 'case_number': 29, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/debian-p0-fts-vset00-00-server-groups_7.0_P1/logs/testrunner-23-Sep-20_21-03-01/test_29'} [2023-09-20 23:48:42,234] - [rest_client:1115] INFO - Making a rest request api=http://172.23.107.9:8091/pools/default verb=POST params=ftsMemoryQuota=3000 client_cert=None verify=False [2023-09-20 23:48:42,266] - [rest_client:3461] INFO - SUCCESS: FTS RAM quota set to 3000mb [2023-09-20 23:48:42,267] - [fts_base:3761] INFO - ==== FTSbasetests setup is started for test #29 test_best_effort_distribution_max_group ==== [2023-09-20 23:48:42,303] - [rest_client:1115] INFO - Making a rest request api=http://172.23.107.9:8091/pools/default verb=POST params=ftsMemoryQuota=3000 client_cert=None verify=False [2023-09-20 23:48:42,342] - [rest_client:3461] INFO - SUCCESS: FTS RAM quota set to 3000mb [2023-09-20 23:48:42,344] - [fts_base:2293] INFO - removing nodes from cluster ... [2023-09-20 23:48:42,360] - [fts_base:2295] INFO - cleanup [ip:172.23.107.9 port:8091 ssh_username:root] [2023-09-20 23:48:42,382] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.107.9, nothing to delete [2023-09-20 23:48:42,396] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:48:42,455] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.107.9:8091 [2023-09-20 23:48:42,455] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:48:42,466] - [cluster_helper:94] INFO - ns_server @ 172.23.107.9:8091 is running [2023-09-20 23:48:42,466] - [fts_base:2318] INFO - Removing user 'cbadminbucket'... [2023-09-20 23:48:42,487] - [rest_client:1079] ERROR - DELETE http://172.23.107.9:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2023-09-20 23:48:42,489] - [fts_base:2322] INFO - b'"User was not found."' [2023-09-20 23:48:42,490] - [fts_base:2324] INFO - Removing user 'clientuser'... [2023-09-20 23:48:42,516] - [rest_client:1079] ERROR - DELETE http://172.23.107.9:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2023-09-20 23:48:42,518] - [fts_base:2328] INFO - b'"User was not found."' [2023-09-20 23:48:42,519] - [fts_base:2232] INFO - Initializing Cluster ... [2023-09-20 23:48:43,285] - [task:156] INFO - server: ip:172.23.107.9 port:8091 ssh_username:root, nodes/self [2023-09-20 23:48:43,297] - [task:161] INFO - {'uptime': '10020', 'memoryTotal': 4068507648, 'memoryFree': 2253754368, 'mcdMemoryReserved': 3104, 'mcdMemoryAllocated': 3104, 'status': 'healthy', 'hostname': '172.23.107.9:8091', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.1-5935-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2079, 'memcached': 11210, 'id': 'ns_1@172.23.107.9', 'ip': '172.23.107.9', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 3880, 'curr_items': 0} [2023-09-20 23:48:43,302] - [rest_client:1254] INFO - pools/default params : memoryQuota=2079 [2023-09-20 23:48:43,326] - [rest_client:1150] INFO - --> in init_cluster...Administrator,password,8091 [2023-09-20 23:48:43,328] - [rest_client:1155] INFO - settings/web params on 172.23.107.9:8091:port=8091&username=Administrator&password=password [2023-09-20 23:48:43,374] - [rest_client:1157] INFO - --> status:True [2023-09-20 23:48:43,384] - [remote_util:302] INFO - SSH Connecting to 172.23.107.9 with username:root, attempt#1 of 5 [2023-09-20 23:48:43,583] - [remote_util:338] INFO - SSH Connected to 172.23.107.9 as root [2023-09-20 23:48:43,745] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:48:44,073] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:48:44,075] - [remote_util:3310] INFO - running command.raw on 172.23.107.9: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2023-09-20 23:48:44,140] - [remote_util:3359] INFO - command executed successfully with root [2023-09-20 23:48:44,140] - [remote_util:5198] INFO - ['ok'] [2023-09-20 23:48:44,151] - [rest_client:1874] INFO - /diag/eval status on 172.23.107.9:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). [2023-09-20 23:48:44,159] - [rest_client:1874] INFO - /diag/eval status on 172.23.107.9:8091: True content: [7,2] command: cluster_compat_mode:get_compat_version(). [2023-09-20 23:48:44,174] - [rest_client:1289] INFO - settings/indexes params : storageMode=plasma index ::0:: node_service ::kv:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: index ::1:: node_service ::kv:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 0 [2023-09-20 23:48:44,241] - [fts_base:2253] INFO - 172.23.107.13 will be configured with services kv index ::2:: node_service ::kv:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 1 [2023-09-20 23:48:44,242] - [fts_base:2253] INFO - 172.23.107.40 will be configured with services kv index ::3:: node_service ::fts:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 2 [2023-09-20 23:48:44,242] - [fts_base:2253] INFO - 172.23.107.41 will be configured with services fts index ::4:: node_service ::fts:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 3 [2023-09-20 23:48:44,243] - [fts_base:2253] INFO - 172.23.104.238 will be configured with services fts index ::5:: node_service ::fts:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 4 [2023-09-20 23:48:44,244] - [fts_base:2253] INFO - 172.23.100.8 will be configured with services fts index ::6:: node_service ::fts:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 5 [2023-09-20 23:48:44,245] - [fts_base:2253] INFO - 172.23.100.10 will be configured with services fts index ::7:: node_service ::fts:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 6 [2023-09-20 23:48:44,246] - [fts_base:2253] INFO - 172.23.100.6 will be configured with services fts index ::8:: node_service ::fts:: cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts', 'fts', 'fts', 'fts']:: available nodes ::[ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.107.41 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.100.6 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root, ip:172.23.120.196 port:8091 ssh_username:root, ip:172.23.104.172 port:8091 ssh_username:root, ip:172.23.104.5 port:8091 ssh_username:root]:: node_num 7 [2023-09-20 23:48:44,246] - [fts_base:2253] INFO - 172.23.106.252 will be configured with services fts [2023-09-20 23:48:45,249] - [task:775] INFO - adding node 172.23.107.13:8091 to cluster [2023-09-20 23:48:45,280] - [rest_client:1619] INFO - adding remote node @172.23.107.13:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:48:55,290] - [rest_client:1957] INFO - rebalance progress took 10.01 seconds [2023-09-20 23:48:55,291] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:49:09,190] - [task:775] INFO - adding node 172.23.107.40:8091 to cluster [2023-09-20 23:49:09,223] - [rest_client:1619] INFO - adding remote node @172.23.107.40:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:49:19,236] - [rest_client:1957] INFO - rebalance progress took 10.01 seconds [2023-09-20 23:49:19,237] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:49:33,161] - [task:775] INFO - adding node 172.23.107.41:8091 to cluster [2023-09-20 23:49:33,200] - [rest_client:1619] INFO - adding remote node @172.23.107.41:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:49:43,220] - [rest_client:1957] INFO - rebalance progress took 10.02 seconds [2023-09-20 23:49:43,220] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:49:57,072] - [task:775] INFO - adding node 172.23.104.238:8091 to cluster [2023-09-20 23:49:57,101] - [rest_client:1619] INFO - adding remote node @172.23.104.238:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:50:07,119] - [rest_client:1957] INFO - rebalance progress took 10.02 seconds [2023-09-20 23:50:07,119] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:50:24,734] - [task:775] INFO - adding node 172.23.100.8:8091 to cluster [2023-09-20 23:50:24,780] - [rest_client:1619] INFO - adding remote node @172.23.100.8:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:50:34,792] - [rest_client:1957] INFO - rebalance progress took 10.01 seconds [2023-09-20 23:50:34,792] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:50:48,651] - [task:775] INFO - adding node 172.23.100.10:8091 to cluster [2023-09-20 23:50:48,702] - [rest_client:1619] INFO - adding remote node @172.23.100.10:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:50:58,716] - [rest_client:1957] INFO - rebalance progress took 10.01 seconds [2023-09-20 23:50:58,716] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:51:12,512] - [task:775] INFO - adding node 172.23.100.6:8091 to cluster [2023-09-20 23:51:12,540] - [rest_client:1619] INFO - adding remote node @172.23.100.6:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:51:22,549] - [rest_client:1957] INFO - rebalance progress took 10.01 seconds [2023-09-20 23:51:22,550] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:51:36,708] - [task:775] INFO - adding node 172.23.106.252:8091 to cluster [2023-09-20 23:51:36,744] - [rest_client:1619] INFO - adding remote node @172.23.106.252:18091 to this cluster @172.23.107.9:8091 [2023-09-20 23:51:46,758] - [rest_client:1957] INFO - rebalance progress took 10.01 seconds [2023-09-20 23:51:46,760] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:52:01,134] - [rest_client:2627] INFO - Node 172.23.100.10 not part of cluster inactiveAdded [2023-09-20 23:52:01,135] - [rest_client:2627] INFO - Node 172.23.100.6 not part of cluster inactiveAdded [2023-09-20 23:52:01,135] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveAdded [2023-09-20 23:52:01,136] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveAdded [2023-09-20 23:52:01,137] - [rest_client:2627] INFO - Node 172.23.106.252 not part of cluster inactiveAdded [2023-09-20 23:52:01,137] - [rest_client:2627] INFO - Node 172.23.107.13 not part of cluster inactiveAdded [2023-09-20 23:52:01,138] - [rest_client:2627] INFO - Node 172.23.107.40 not part of cluster inactiveAdded [2023-09-20 23:52:01,138] - [rest_client:2627] INFO - Node 172.23.107.41 not part of cluster inactiveAdded [2023-09-20 23:52:01,161] - [rest_client:1851] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.10,ns_1@172.23.100.6,ns_1@172.23.100.8,ns_1@172.23.104.238,ns_1@172.23.106.252,ns_1@172.23.107.13,ns_1@172.23.107.40,ns_1@172.23.107.41,ns_1@172.23.107.9', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2023-09-20 23:52:01,216] - [rest_client:1856] INFO - rebalance operation started [2023-09-20 23:52:11,226] - [task:845] INFO - Rebalance - status: none, progress: 100.00% [2023-09-20 23:52:11,252] - [task:904] INFO - rebalancing was completed with progress: 100% in 10.034680128097534 sec [2023-09-20 23:52:11,280] - [remote_util:302] INFO - SSH Connecting to 172.23.107.9 with username:root, attempt#1 of 5 [2023-09-20 23:52:11,459] - [remote_util:338] INFO - SSH Connected to 172.23.107.9 as root [2023-09-20 23:52:11,601] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:52:11,923] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:52:11,926] - [remote_util:3310] INFO - running command.raw on 172.23.107.9: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2023-09-20 23:52:11,993] - [remote_util:3359] INFO - command executed successfully with root [2023-09-20 23:52:11,994] - [remote_util:5198] INFO - ['ok'] [2023-09-20 23:52:11,995] - [fts_base:4135] INFO - Enabled diag/eval for non-local hosts from 172.23.107.9 [2023-09-20 23:52:12,067] - [rest_client:1079] ERROR - DELETE http://172.23.107.9:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2023-09-20 23:52:12,070] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2023-09-20 23:52:12,232] - [fts_base:4047] INFO - updating bleve_max_result_window of node : ip:172.23.100.10 port:8091 ssh_username:root [2023-09-20 23:52:12,252] - [rest_client:3603] INFO - {"bleveMaxResultWindow": "100000000"} [2023-09-20 23:52:12,254] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/managerOptions verb=PUT params={"bleveMaxResultWindow": "100000000"} client_cert=None verify=False [2023-09-20 23:52:12,272] - [rest_client:3608] INFO - Updated bleveMaxResultWindow [2023-09-20 23:52:12,272] - [fts_base:4047] INFO - updating bleve_max_result_window of node : ip:172.23.100.6 port:8091 ssh_username:root [2023-09-20 23:52:12,288] - [rest_client:3603] INFO - {"bleveMaxResultWindow": "100000000"} [2023-09-20 23:52:12,289] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.6:8094/api/managerOptions verb=PUT params={"bleveMaxResultWindow": "100000000"} client_cert=None verify=False [2023-09-20 23:52:12,304] - [rest_client:3608] INFO - Updated bleveMaxResultWindow [2023-09-20 23:52:12,305] - [fts_base:4047] INFO - updating bleve_max_result_window of node : ip:172.23.100.8 port:8091 ssh_username:root [2023-09-20 23:52:12,322] - [rest_client:3603] INFO - {"bleveMaxResultWindow": "100000000"} [2023-09-20 23:52:12,322] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.8:8094/api/managerOptions verb=PUT params={"bleveMaxResultWindow": "100000000"} client_cert=None verify=False [2023-09-20 23:52:12,338] - [rest_client:3608] INFO - Updated bleveMaxResultWindow [2023-09-20 23:52:12,339] - [fts_base:4047] INFO - updating bleve_max_result_window of node : ip:172.23.104.238 port:8091 ssh_username:root [2023-09-20 23:52:12,358] - [rest_client:3603] INFO - {"bleveMaxResultWindow": "100000000"} [2023-09-20 23:52:12,359] - [rest_client:1115] INFO - Making a rest request api=http://172.23.104.238:8094/api/managerOptions verb=PUT params={"bleveMaxResultWindow": "100000000"} client_cert=None verify=False [2023-09-20 23:52:12,378] - [rest_client:3608] INFO - Updated bleveMaxResultWindow [2023-09-20 23:52:12,379] - [fts_base:4047] INFO - updating bleve_max_result_window of node : ip:172.23.106.252 port:8091 ssh_username:root [2023-09-20 23:52:12,392] - [rest_client:3603] INFO - {"bleveMaxResultWindow": "100000000"} [2023-09-20 23:52:12,393] - [rest_client:1115] INFO - Making a rest request api=http://172.23.106.252:8094/api/managerOptions verb=PUT params={"bleveMaxResultWindow": "100000000"} client_cert=None verify=False [2023-09-20 23:52:12,407] - [rest_client:3608] INFO - Updated bleveMaxResultWindow [2023-09-20 23:52:12,408] - [fts_base:4047] INFO - updating bleve_max_result_window of node : ip:172.23.107.41 port:8091 ssh_username:root [2023-09-20 23:52:12,425] - [rest_client:3603] INFO - {"bleveMaxResultWindow": "100000000"} [2023-09-20 23:52:12,426] - [rest_client:1115] INFO - Making a rest request api=http://172.23.107.41:8094/api/managerOptions verb=PUT params={"bleveMaxResultWindow": "100000000"} client_cert=None verify=False [2023-09-20 23:52:12,447] - [rest_client:3608] INFO - Updated bleveMaxResultWindow [2023-09-20 23:52:13,314] - [rest_client:3005] INFO - http://172.23.107.9:8091/pools/default/buckets with param: name=default&ramQuotaMB=829&replicaNumber=2&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2023-09-20 23:52:13,356] - [rest_client:3030] INFO - 0.04 seconds to create bucket default [2023-09-20 23:52:13,357] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.107.9 to accept set ops [2023-09-20 23:52:16,407] - [data_helper:308] INFO - creating direct client 172.23.107.13:11210 default [2023-09-20 23:52:16,792] - [data_helper:308] INFO - creating direct client 172.23.107.9:11210 default [2023-09-20 23:52:16,986] - [data_helper:308] INFO - creating direct client 172.23.107.40:11210 default [2023-09-20 23:52:17,541] - [data_helper:308] INFO - creating direct client 172.23.107.13:11210 default [2023-09-20 23:52:17,752] - [data_helper:308] INFO - creating direct client 172.23.107.9:11210 default [2023-09-20 23:52:17,938] - [data_helper:308] INFO - creating direct client 172.23.107.40:11210 default [2023-09-20 23:52:19,238] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 829 [2023-09-20 23:52:19,246] - [fts_base:3807] INFO - ==== FTSbasetests setup is finished for test #29 test_best_effort_distribution_max_group ==== [2023-09-20 23:52:19,387] - [rest_client:4333] INFO - zone sg1 is deleted [2023-09-20 23:52:19,412] - [rest_client:4368] INFO - zone sg3 is existed but no node in it [2023-09-20 23:52:19,466] - [rest_client:4333] INFO - zone sg3 is deleted [2023-09-20 23:52:19,484] - [rest_client:4368] INFO - zone sg2 is existed but no node in it [2023-09-20 23:52:19,533] - [rest_client:4333] INFO - zone sg2 is deleted [2023-09-20 23:52:19,642] - [rest_client:4314] INFO - zone sg1 is added [2023-09-20 23:52:19,734] - [rest_client:4314] INFO - zone sg2 is added [2023-09-20 23:52:19,826] - [rest_client:4314] INFO - zone sg3 is added [2023-09-20 23:52:20,367] - [data_helper:308] INFO - creating direct client 172.23.107.13:11210 default [2023-09-20 23:52:20,730] - [data_helper:308] INFO - creating direct client 172.23.107.9:11210 default [2023-09-20 23:52:20,887] - [data_helper:308] INFO - creating direct client 172.23.107.40:11210 default [2023-09-20 23:52:22,306] - [fts_base:5532] INFO - Loading phase complete! [2023-09-20 23:52:22,712] - [fts_base:1227] INFO - Checking if index already exists ... [2023-09-20 23:52:22,771] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.6:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:23,004] - [rest_client:1139] ERROR - b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' [2023-09-20 23:52:23,006] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.6:8094/api/index/fts_idx verb=DELETE params= client_cert=None verify=False [2023-09-20 23:52:23,067] - [rest_client:1139] ERROR - b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' [2023-09-20 23:52:23,068] - [fts_base:1236] INFO - Creating fulltext-index fts_idx on 172.23.100.6 [2023-09-20 23:52:23,070] - [rest_client:3501] INFO - {"type": "fulltext-index", "name": "fts_idx", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}} [2023-09-20 23:52:23,071] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.6:8094/api/index/fts_idx verb=PUT params={"type": "fulltext-index", "name": "fts_idx", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171, "indexPartitions": 1}, "sourceParams": {}} client_cert=None verify=False [2023-09-20 23:52:23,523] - [rest_client:3504] INFO - Index fts_idx created [2023-09-20 23:52:25,572] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.8:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:25,588] - [fts_base:4823] INFO - Validating index distribution for fts_idx ... [2023-09-20 23:52:25,624] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:25,678] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:25,698] - [fts_base:4841] INFO - Validated: Number of PIndexes = 1 [2023-09-20 23:52:25,715] - [fts_base:4853] INFO - Validated: Every pIndex serves 1024 partitions or lesser [2023-09-20 23:52:25,716] - [fts_base:4878] INFO - Expecting num of partitions in each node in range -854-1024 [2023-09-20 23:52:25,716] - [fts_base:4893] INFO - Validated: Node e97092b5fe733f460453f0bd0e2293d8 houses 1 pindexes which serve 1024 partitions [2023-09-20 23:52:25,748] - [rest_client:1115] INFO - Making a rest request api=http://172.23.107.41:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:25,797] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.6:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:25,842] - [fts_base:1260] INFO - Updating fulltext-index fts_idx on 172.23.100.8 [2023-09-20 23:52:25,843] - [rest_client:3511] INFO - { "type": "fulltext-index", "name": "fts_idx", "uuid": "429733fe88af16e9", "sourceType": "gocbcore", "sourceName": "default", "sourceUUID": "7ce8e6708e174a8a87ef6ae37e491927", "planParams": { "maxPartitionsPerPIndex": 1024, "indexPartitions": 12 }, "params": { "doc_config": { "docid_prefix_delim": "", "docid_regexp": "", "mode": "type_field", "type_field": "type" }, "mapping": { "analysis": {}, "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "default_mapping": { "dynamic": true, "enabled": true }, "default_type": "_default", "docvalues_dynamic": true, "index_dynamic": true, "store_dynamic": false, "type_field": "_type" }, "store": { "indexType": "scorch", "mossStoreOptions": {}, "segmentVersion": 15 } }, "sourceParams": {} } [2023-09-20 23:52:25,844] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.8:8094/api/index/fts_idx verb=PUT params={"type": "fulltext-index", "name": "fts_idx", "uuid": "429733fe88af16e9", "sourceType": "gocbcore", "sourceName": "default", "sourceUUID": "7ce8e6708e174a8a87ef6ae37e491927", "planParams": {"maxPartitionsPerPIndex": 1024, "indexPartitions": 12}, "params": {"doc_config": {"docid_prefix_delim": "", "docid_regexp": "", "mode": "type_field", "type_field": "type"}, "mapping": {"analysis": {}, "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "default_mapping": {"dynamic": true, "enabled": true}, "default_type": "_default", "docvalues_dynamic": true, "index_dynamic": true, "store_dynamic": false, "type_field": "_type"}, "store": {"indexType": "scorch", "mossStoreOptions": {}, "segmentVersion": 15}}, "sourceParams": {}} client_cert=None verify=False [2023-09-20 23:52:26,080] - [rest_client:3514] INFO - Index/alias fts_idx updated [2023-09-20 23:52:26,149] - [rest_client:1115] INFO - Making a rest request api=http://172.23.106.252:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:26,223] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.8:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:26,317] - [fts_base:1260] INFO - Updating fulltext-index fts_idx on 172.23.104.238 [2023-09-20 23:52:26,318] - [rest_client:3511] INFO - { "type": "fulltext-index", "name": "fts_idx", "uuid": "13a919c7a80d21fe", "sourceType": "gocbcore", "sourceName": "default", "sourceUUID": "7ce8e6708e174a8a87ef6ae37e491927", "planParams": { "maxPartitionsPerPIndex": 1024, "indexPartitions": 12, "numReplicas": 2 }, "params": { "doc_config": { "docid_prefix_delim": "", "docid_regexp": "", "mode": "type_field", "type_field": "type" }, "mapping": { "analysis": {}, "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "default_mapping": { "dynamic": true, "enabled": true }, "default_type": "_default", "docvalues_dynamic": true, "index_dynamic": true, "store_dynamic": false, "type_field": "_type" }, "store": { "indexType": "scorch", "mossStoreOptions": {}, "segmentVersion": 15 } }, "sourceParams": {} } [2023-09-20 23:52:26,319] - [rest_client:1115] INFO - Making a rest request api=http://172.23.104.238:8094/api/index/fts_idx verb=PUT params={"type": "fulltext-index", "name": "fts_idx", "uuid": "13a919c7a80d21fe", "sourceType": "gocbcore", "sourceName": "default", "sourceUUID": "7ce8e6708e174a8a87ef6ae37e491927", "planParams": {"maxPartitionsPerPIndex": 1024, "indexPartitions": 12, "numReplicas": 2}, "params": {"doc_config": {"docid_prefix_delim": "", "docid_regexp": "", "mode": "type_field", "type_field": "type"}, "mapping": {"analysis": {}, "default_analyzer": "standard", "default_datetime_parser": "dateTimeOptional", "default_field": "_all", "default_mapping": {"dynamic": true, "enabled": true}, "default_type": "_default", "docvalues_dynamic": true, "index_dynamic": true, "store_dynamic": false, "type_field": "_type"}, "store": {"indexType": "scorch", "mossStoreOptions": {}, "segmentVersion": 15}}, "sourceParams": {}} client_cert=None verify=False [2023-09-20 23:52:26,570] - [rest_client:3514] INFO - Index/alias fts_idx updated [2023-09-20 23:52:26,649] - [rest_client:1115] INFO - Making a rest request api=http://172.23.106.252:8094/api/index/fts_idx verb=GET params= client_cert=None verify=False [2023-09-20 23:52:26,707] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/index/fts_idx/count verb=GET params= client_cert=None verify=False [2023-09-20 23:52:26,771] - [rest_client:2541] INFO - http://172.23.107.9:8091/pools/default/buckets/default/stats?zoom=minute [2023-09-20 23:52:26,883] - [fts_base:4700] INFO - Docs in bucket = 665, docs in FTS index 'fts_idx': 541 [2023-09-20 23:52:32,945] - [rest_client:1115] INFO - Making a rest request api=http://172.23.107.41:8094/api/index/fts_idx/count verb=GET params= client_cert=None verify=False [2023-09-20 23:52:32,988] - [rest_client:2541] INFO - http://172.23.107.9:8091/pools/default/buckets/default/stats?zoom=minute [2023-09-20 23:52:33,085] - [fts_base:4700] INFO - Docs in bucket = 1000, docs in FTS index 'fts_idx': 1000 [2023-09-20 23:52:33,128] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.8:8094/api/index/fts_idx/count verb=GET params= client_cert=None verify=False [2023-09-20 23:52:33,174] - [rest_client:2541] INFO - http://172.23.107.9:8091/pools/default/buckets/default/stats?zoom=minute [2023-09-20 23:52:33,264] - [fts_base:4700] INFO - Docs in bucket = 1000, docs in FTS index 'fts_idx': 1000 [2023-09-20 23:52:33,310] - [fts_base:2679] INFO - Running query {"indexName": "fts_idx", "size": 10000000, "from": 0, "explain": false, "query": {"match": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.100.10 : Administrator: [2023-09-20 23:52:33,311] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/index/fts_idx/query verb=POST params=b'{"indexName": "fts_idx", "size": 10000000, "from": 0, "explain": false, "query": {"match": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False [2023-09-20 23:52:33,451] - [fts_base:3520] INFO - Starting failover for nodes:[ip:172.23.106.252 port:8091 ssh_username:root] at C1 cluster 172.23.107.9 [2023-09-20 23:52:34,362] - [task:4120] INFO - Failing over 172.23.106.252:8091 with graceful=False [2023-09-20 23:52:35,319] - [rest_client:1796] INFO - fail_over node ns_1@172.23.106.252 successful [2023-09-20 23:52:35,320] - [task:4100] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2023-09-20 23:52:35,321] - [fts_base:3520] INFO - Starting failover for nodes:[ip:172.23.107.41 port:8091 ssh_username:root] at C1 cluster 172.23.107.9 [2023-09-20 23:52:36,364] - [rest_client:2627] INFO - Node 172.23.106.252 not part of cluster inactiveFailed [2023-09-20 23:52:36,387] - [task:4120] INFO - Failing over 172.23.107.41:8091 with graceful=False [2023-09-20 23:52:37,197] - [rest_client:1796] INFO - fail_over node ns_1@172.23.107.41 successful [2023-09-20 23:52:37,198] - [task:4100] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2023-09-20 23:52:37,336] - [rest_client:2627] INFO - Node 172.23.106.252 not part of cluster inactiveFailed [2023-09-20 23:52:37,337] - [rest_client:2627] INFO - Node 172.23.107.41 not part of cluster inactiveFailed [2023-09-20 23:52:37,353] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.8:8094/api/index/fts_idx/count verb=GET params= client_cert=None verify=False [2023-09-20 23:52:37,384] - [rest_client:2541] INFO - http://172.23.107.9:8091/pools/default/buckets/default/stats?zoom=minute [2023-09-20 23:52:37,501] - [fts_base:4700] INFO - Docs in bucket = 1000, docs in FTS index 'fts_idx': 1000 [2023-09-20 23:52:37,508] - [rest_client:1874] INFO - /diag/eval status on 172.23.107.9:8091: True content: true command: ns_orchestrator:needs_rebalance() [2023-09-20 23:52:38,208] - [rest_client:2627] INFO - Node 172.23.106.252 not part of cluster inactiveFailed [2023-09-20 23:52:38,210] - [rest_client:2627] INFO - Node 172.23.107.41 not part of cluster inactiveFailed [2023-09-20 23:52:38,236] - [rest_client:1851] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.10,ns_1@172.23.100.6,ns_1@172.23.100.8,ns_1@172.23.104.238,ns_1@172.23.106.252,ns_1@172.23.107.13,ns_1@172.23.107.40,ns_1@172.23.107.41,ns_1@172.23.107.9', 'ejectedNodes': 'ns_1@172.23.107.41', 'user': 'Administrator', 'password': 'password'} [2023-09-20 23:52:38,294] - [rest_client:1856] INFO - rebalance operation started [2023-09-20 23:52:48,302] - [rest_client:2018] INFO - rebalance percentage : 19.00 % [2023-09-20 23:52:48,302] - [task:845] INFO - Rebalance - status: running, progress: 19.00% [2023-09-20 23:53:08,333] - [task:845] INFO - Rebalance - status: none, progress: 100.00% [2023-09-20 23:53:08,365] - [task:904] INFO - rebalancing was completed with progress: 100% in 30.069478750228882 sec [2023-09-20 23:53:08,390] - [rest_client:4368] INFO - zone Group 1 is existed but no node in it [2023-09-20 23:53:08,669] - [fts_base:3520] INFO - Starting failover for nodes:[ip:172.23.100.8 port:8091 ssh_username:root] at C1 cluster 172.23.107.9 [2023-09-20 23:53:09,417] - [task:4120] INFO - Failing over 172.23.100.8:8091 with graceful=False [2023-09-20 23:53:10,488] - [rest_client:1796] INFO - fail_over node ns_1@172.23.100.8 successful [2023-09-20 23:53:10,489] - [task:4100] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2023-09-20 23:53:10,491] - [fts_base:3520] INFO - Starting failover for nodes:[ip:172.23.104.238 port:8091 ssh_username:root] at C1 cluster 172.23.107.9 [2023-09-20 23:53:11,517] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:11,533] - [task:4120] INFO - Failing over 172.23.104.238:8091 with graceful=False [2023-09-20 23:53:12,331] - [rest_client:1796] INFO - fail_over node ns_1@172.23.104.238 successful [2023-09-20 23:53:12,331] - [task:4100] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2023-09-20 23:53:12,474] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:12,475] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveFailed [2023-09-20 23:53:12,492] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:12,493] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveFailed [2023-09-20 23:53:12,533] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:12,534] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveFailed [2023-09-20 23:53:12,566] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:12,567] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveFailed [2023-09-20 23:53:12,617] - [fts_base:3520] INFO - Starting failover for nodes:[ip:172.23.100.6 port:8091 ssh_username:root] at C1 cluster 172.23.107.9 [2023-09-20 23:53:13,353] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:13,354] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveFailed [2023-09-20 23:53:13,373] - [task:4120] INFO - Failing over 172.23.100.6:8091 with graceful=False [2023-09-20 23:53:14,407] - [rest_client:1796] INFO - fail_over node ns_1@172.23.100.6 successful [2023-09-20 23:53:14,408] - [task:4100] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2023-09-20 23:53:14,420] - [rest_client:1874] INFO - /diag/eval status on 172.23.107.9:8091: True content: true command: ns_orchestrator:needs_rebalance() [2023-09-20 23:53:15,430] - [rest_client:2627] INFO - Node 172.23.100.6 not part of cluster inactiveFailed [2023-09-20 23:53:15,431] - [rest_client:2627] INFO - Node 172.23.100.8 not part of cluster inactiveFailed [2023-09-20 23:53:15,432] - [rest_client:2627] INFO - Node 172.23.104.238 not part of cluster inactiveFailed [2023-09-20 23:53:15,461] - [rest_client:1851] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.10,ns_1@172.23.100.6,ns_1@172.23.100.8,ns_1@172.23.104.238,ns_1@172.23.107.13,ns_1@172.23.107.40,ns_1@172.23.107.9', 'ejectedNodes': 'ns_1@172.23.100.6', 'user': 'Administrator', 'password': 'password'} [2023-09-20 23:53:15,521] - [rest_client:1856] INFO - rebalance operation started [2023-09-20 23:53:25,529] - [rest_client:2018] INFO - rebalance percentage : 58.00 % [2023-09-20 23:53:25,530] - [task:845] INFO - Rebalance - status: running, progress: 58.00% [2023-09-20 23:53:45,555] - [rest_client:2018] INFO - rebalance percentage : 83.00 % [2023-09-20 23:53:45,555] - [task:845] INFO - Rebalance - status: running, progress: 83.00% [2023-09-20 23:54:05,583] - [task:845] INFO - Rebalance - status: none, progress: 100.00% [2023-09-20 23:54:05,614] - [task:904] INFO - rebalancing was completed with progress: 100% in 50.09091830253601 sec [2023-09-20 23:54:05,632] - [fts_base:2679] INFO - Running query {"indexName": "fts_idx", "size": 10000000, "from": 0, "explain": false, "query": {"match": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.100.10 : Administrator: [2023-09-20 23:54:05,633] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/index/fts_idx/query verb=POST params=b'{"indexName": "fts_idx", "size": 10000000, "from": 0, "explain": false, "query": {"match": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False [2023-09-20 23:54:05,662] - [fts_base:3992] INFO - ==== FTSbasetests cleanup is started for test #29 test_best_effort_distribution_max_group ==== [2023-09-20 23:54:05,696] - [fts_base:1330] INFO - Deleting fulltext-index fts_idx on 172.23.100.10 [2023-09-20 23:54:05,697] - [rest_client:1115] INFO - Making a rest request api=http://172.23.100.10:8094/api/index/fts_idx verb=DELETE params= client_cert=None verify=False [2023-09-20 23:54:05,837] - [remote_util:302] INFO - SSH Connecting to 172.23.100.10 with username:root, attempt#1 of 5 [2023-09-20 23:54:06,025] - [remote_util:338] INFO - SSH Connected to 172.23.100.10 as root [2023-09-20 23:54:06,285] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:06,615] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:54:06,619] - [remote_util:3310] INFO - running command.raw on 172.23.100.10: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep ^fts_idx | wc -l [2023-09-20 23:54:06,671] - [remote_util:3359] INFO - command executed successfully with root [2023-09-20 23:54:06,673] - [fts_base:2194] INFO - 0 [2023-09-20 23:54:08,674] - [fts_base:1342] INFO - Validated: all index files for fts_idx deleted from disk [2023-09-20 23:54:08,677] - [fts_base:2293] INFO - removing nodes from cluster ... [2023-09-20 23:54:08,694] - [fts_base:2295] INFO - cleanup [ip:172.23.107.9 port:8091 ssh_username:root, ip:172.23.107.13 port:8091 ssh_username:root, ip:172.23.107.40 port:8091 ssh_username:root, ip:172.23.104.238 port:8091 ssh_username:root, ip:172.23.100.8 port:8091 ssh_username:root, ip:172.23.100.10 port:8091 ssh_username:root, ip:172.23.106.252 port:8091 ssh_username:root] [2023-09-20 23:54:08,750] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.107.9 [2023-09-20 23:54:10,018] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2023-09-20 23:54:10,026] - [rest_client:140] INFO - node 172.23.107.9 existing buckets : [] [2023-09-20 23:54:10,027] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.107.9 [2023-09-20 23:54:10,042] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:10,098] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2023-09-20 23:54:10,104] - [rest_client:1851] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.100.10,ns_1@172.23.107.13,ns_1@172.23.107.40,ns_1@172.23.107.9', 'ejectedNodes': 'ns_1@172.23.100.10,ns_1@172.23.107.13,ns_1@172.23.107.40', 'user': 'Administrator', 'password': 'password'} [2023-09-20 23:54:10,161] - [rest_client:1856] INFO - rebalance operation started [2023-09-20 23:54:10,168] - [rest_client:2018] INFO - rebalance percentage : 0.00 % [2023-09-20 23:54:20,174] - [rest_client:2018] INFO - rebalance percentage : 66.00 % [2023-09-20 23:54:40,186] - [rest_client:1957] INFO - rebalance progress took 30.02 seconds [2023-09-20 23:54:40,186] - [rest_client:1958] INFO - sleep for 10 seconds after rebalance... [2023-09-20 23:54:50,220] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.107.9 port:8091 ssh_username:root ? [('ns_1@172.23.100.10', 8091), ('ns_1@172.23.107.13', 8091), ('ns_1@172.23.107.40', 8091)] [2023-09-20 23:54:50,232] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.107.9:8091 [2023-09-20 23:54:50,232] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,242] - [cluster_helper:94] INFO - ns_server @ 172.23.107.9:8091 is running [2023-09-20 23:54:50,253] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.107.13, nothing to delete [2023-09-20 23:54:50,259] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,267] - [rest_client:1079] ERROR - GET http://172.23.107.13:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2023-09-20 23:54:50,284] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.107.13:8091 [2023-09-20 23:54:50,285] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,290] - [cluster_helper:94] INFO - ns_server @ 172.23.107.13:8091 is running [2023-09-20 23:54:50,300] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.107.40, nothing to delete [2023-09-20 23:54:50,305] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,315] - [rest_client:1079] ERROR - GET http://172.23.107.40:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2023-09-20 23:54:50,333] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.107.40:8091 [2023-09-20 23:54:50,334] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,340] - [cluster_helper:94] INFO - ns_server @ 172.23.107.40:8091 is running [2023-09-20 23:54:50,357] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.238, nothing to delete [2023-09-20 23:54:50,364] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,376] - [rest_client:1079] ERROR - GET http://172.23.104.238:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2023-09-20 23:54:50,399] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.238:8091 [2023-09-20 23:54:50,400] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,406] - [cluster_helper:94] INFO - ns_server @ 172.23.104.238:8091 is running [2023-09-20 23:54:50,417] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.100.8, nothing to delete [2023-09-20 23:54:50,423] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,432] - [rest_client:1079] ERROR - GET http://172.23.100.8:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2023-09-20 23:54:50,452] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.100.8:8091 [2023-09-20 23:54:50,453] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,461] - [cluster_helper:94] INFO - ns_server @ 172.23.100.8:8091 is running [2023-09-20 23:54:50,472] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.100.10, nothing to delete [2023-09-20 23:54:50,479] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,489] - [rest_client:1079] ERROR - GET http://172.23.100.10:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2023-09-20 23:54:50,504] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.100.10:8091 [2023-09-20 23:54:50,505] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,510] - [cluster_helper:94] INFO - ns_server @ 172.23.100.10:8091 is running [2023-09-20 23:54:50,523] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.252, nothing to delete [2023-09-20 23:54:50,530] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,540] - [rest_client:1079] ERROR - GET http://172.23.106.252:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2023-09-20 23:54:50,554] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.252:8091 [2023-09-20 23:54:50,555] - [rest_client:44] INFO - -->is_ns_server_running? [2023-09-20 23:54:50,562] - [cluster_helper:94] INFO - ns_server @ 172.23.106.252:8091 is running Cluster instance shutdown with force [2023-09-20 23:54:50,562] - [fts_base:2318] INFO - Removing user 'cbadminbucket'... [2023-09-20 23:54:50,595] - [fts_base:2324] INFO - Removing user 'clientuser'... [2023-09-20 23:54:50,619] - [rest_client:1079] ERROR - DELETE http://172.23.107.9:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2023-09-20 23:54:50,621] - [fts_base:2328] INFO - b'"User was not found."' [2023-09-20 23:54:50,621] - [ntonencryptionBase:144] INFO - Disable up node to node encryption - status = disable and clusterEncryptionLevel = control [2023-09-20 23:54:50,633] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,642] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,651] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,665] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,676] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,687] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,697] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,710] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,722] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,735] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,748] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,761] - [rest_client:3115] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2023-09-20 23:54:50,766] - [ntonencryptionBase:102] INFO - Changing encryption Level - clusterEncryptionLevel = control [2023-09-20 23:54:50,772] - [remote_util:302] INFO - SSH Connecting to 172.23.107.9 with username:root, attempt#1 of 5 [2023-09-20 23:54:50,950] - [remote_util:338] INFO - SSH Connected to 172.23.107.9 as root [2023-09-20 23:54:51,304] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:51,587] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:54:51,958] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:54:52,353] - [remote_util:302] INFO - SSH Connecting to 172.23.107.13 with username:root, attempt#1 of 5 [2023-09-20 23:54:52,533] - [remote_util:338] INFO - SSH Connected to 172.23.107.13 as root [2023-09-20 23:54:52,681] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:53,225] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:54:53,612] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:54:54,009] - [remote_util:302] INFO - SSH Connecting to 172.23.107.40 with username:root, attempt#1 of 5 [2023-09-20 23:54:54,184] - [remote_util:338] INFO - SSH Connected to 172.23.107.40 as root [2023-09-20 23:54:54,346] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:54,676] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:54:55,224] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:54:55,629] - [remote_util:302] INFO - SSH Connecting to 172.23.107.41 with username:root, attempt#1 of 5 [2023-09-20 23:54:55,768] - [remote_util:338] INFO - SSH Connected to 172.23.107.41 as root [2023-09-20 23:54:56,211] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:56,542] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:54:56,906] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:54:57,307] - [remote_util:302] INFO - SSH Connecting to 172.23.104.238 with username:root, attempt#1 of 5 [2023-09-20 23:54:57,576] - [remote_util:338] INFO - SSH Connected to 172.23.104.238 as root [2023-09-20 23:54:57,759] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:58,124] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:54:58,490] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:54:59,282] - [remote_util:302] INFO - SSH Connecting to 172.23.100.8 with username:root, attempt#1 of 5 [2023-09-20 23:54:59,459] - [remote_util:338] INFO - SSH Connected to 172.23.100.8 as root [2023-09-20 23:54:59,621] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:54:59,961] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:00,540] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:00,929] - [remote_util:302] INFO - SSH Connecting to 172.23.100.10 with username:root, attempt#1 of 5 [2023-09-20 23:55:01,110] - [remote_util:338] INFO - SSH Connected to 172.23.100.10 as root [2023-09-20 23:55:01,269] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:01,599] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:01,974] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:02,355] - [remote_util:302] INFO - SSH Connecting to 172.23.100.6 with username:root, attempt#1 of 5 [2023-09-20 23:55:02,537] - [remote_util:338] INFO - SSH Connected to 172.23.100.6 as root [2023-09-20 23:55:02,770] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:03,096] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:03,467] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:03,848] - [remote_util:302] INFO - SSH Connecting to 172.23.106.252 with username:root, attempt#1 of 5 [2023-09-20 23:55:04,030] - [remote_util:338] INFO - SSH Connected to 172.23.106.252 as root [2023-09-20 23:55:04,253] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:04,572] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:04,946] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:05,332] - [remote_util:302] INFO - SSH Connecting to 172.23.120.196 with username:root, attempt#1 of 5 [2023-09-20 23:55:05,558] - [remote_util:338] INFO - SSH Connected to 172.23.120.196 as root [2023-09-20 23:55:05,902] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:06,223] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:06,477] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:07,240] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2023-09-20 23:55:07,427] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2023-09-20 23:55:07,700] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:08,048] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:08,437] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:09,005] - [remote_util:302] INFO - SSH Connecting to 172.23.104.5 with username:root, attempt#1 of 5 [2023-09-20 23:55:09,070] - [remote_util:338] INFO - SSH Connected to 172.23.104.5 as root [2023-09-20 23:55:09,841] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: ubuntu 20.04, is_linux_distro: True [2023-09-20 23:55:10,138] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: ubuntu 20.04 [2023-09-20 23:55:10,454] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control [2023-09-20 23:55:10,745] - [ntonencryptionBase:126] INFO - Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] [2023-09-20 23:55:10,747] - [ntonencryptionBase:127] INFO - Error of setting-security command is [] [2023-09-20 23:55:10,749] - [ntonencryptionBase:40] INFO - Changing node-to-node-encryption to disable [2023-09-20 23:55:10,760] - [remote_util:302] INFO - SSH Connecting to 172.23.107.9 with username:root, attempt#1 of 5 [2023-09-20 23:55:10,942] - [remote_util:338] INFO - SSH Connected to 172.23.107.9 as root [2023-09-20 23:55:11,095] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:11,417] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:11,790] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:12,248] - [remote_util:302] INFO - SSH Connecting to 172.23.107.13 with username:root, attempt#1 of 5 [2023-09-20 23:55:12,425] - [remote_util:338] INFO - SSH Connected to 172.23.107.13 as root [2023-09-20 23:55:12,570] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:12,891] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:13,247] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:13,661] - [remote_util:302] INFO - SSH Connecting to 172.23.107.40 with username:root, attempt#1 of 5 [2023-09-20 23:55:13,836] - [remote_util:338] INFO - SSH Connected to 172.23.107.40 as root [2023-09-20 23:55:13,999] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:14,326] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:14,637] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:15,037] - [remote_util:302] INFO - SSH Connecting to 172.23.107.41 with username:root, attempt#1 of 5 [2023-09-20 23:55:15,216] - [remote_util:338] INFO - SSH Connected to 172.23.107.41 as root [2023-09-20 23:55:15,445] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:15,768] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:16,321] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:16,737] - [remote_util:302] INFO - SSH Connecting to 172.23.104.238 with username:root, attempt#1 of 5 [2023-09-20 23:55:16,965] - [remote_util:338] INFO - SSH Connected to 172.23.104.238 as root [2023-09-20 23:55:17,133] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:17,466] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:17,857] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:18,759] - [remote_util:302] INFO - SSH Connecting to 172.23.100.8 with username:root, attempt#1 of 5 [2023-09-20 23:55:18,940] - [remote_util:338] INFO - SSH Connected to 172.23.100.8 as root [2023-09-20 23:55:19,100] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:19,436] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:19,807] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:20,215] - [remote_util:302] INFO - SSH Connecting to 172.23.100.10 with username:root, attempt#1 of 5 [2023-09-20 23:55:20,389] - [remote_util:338] INFO - SSH Connected to 172.23.100.10 as root [2023-09-20 23:55:20,543] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:20,873] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:21,359] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:21,775] - [remote_util:302] INFO - SSH Connecting to 172.23.100.6 with username:root, attempt#1 of 5 [2023-09-20 23:55:21,955] - [remote_util:338] INFO - SSH Connected to 172.23.100.6 as root [2023-09-20 23:55:22,182] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:22,512] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:22,894] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:23,290] - [remote_util:302] INFO - SSH Connecting to 172.23.106.252 with username:root, attempt#1 of 5 [2023-09-20 23:55:23,430] - [remote_util:338] INFO - SSH Connected to 172.23.106.252 as root [2023-09-20 23:55:23,666] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:24,009] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:24,397] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:24,816] - [remote_util:302] INFO - SSH Connecting to 172.23.120.196 with username:root, attempt#1 of 5 [2023-09-20 23:55:25,047] - [remote_util:338] INFO - SSH Connected to 172.23.120.196 as root [2023-09-20 23:55:25,394] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:25,710] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:26,005] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:26,920] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2023-09-20 23:55:27,100] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2023-09-20 23:55:27,373] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: debian 11, is_linux_distro: True [2023-09-20 23:55:27,712] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: debian 11 [2023-09-20 23:55:28,111] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:28,719] - [remote_util:302] INFO - SSH Connecting to 172.23.104.5 with username:root, attempt#1 of 5 [2023-09-20 23:55:28,780] - [remote_util:338] INFO - SSH Connected to 172.23.104.5 as root [2023-09-20 23:55:29,564] - [remote_util:3474] INFO - os_distro: Ubuntu, os_version: ubuntu 20.04, is_linux_distro: True [2023-09-20 23:55:29,815] - [remote_util:3639] INFO - extract_remote_info-->distribution_type: Ubuntu, distribution_version: ubuntu 20.04 [2023-09-20 23:55:30,124] - [remote_util:4670] INFO - command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable [2023-09-20 23:55:30,432] - [ntonencryptionBase:63] INFO - Output of node-to-node-encryption command is ['SUCCESS: Switched node-to-node encryption off'] [2023-09-20 23:55:30,433] - [ntonencryptionBase:64] INFO - Error of node-to-node-encryption command is [] [2023-09-20 23:55:30,433] - [fts_base:4004] INFO - ==== FTSbasetests cleanup is finished for test #29 test_best_effort_distribution_max_group === [2023-09-20 23:55:30,434] - [fts_base:4006] INFO - closing all ssh connections [2023-09-20 23:55:30,439] - [fts_base:4010] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite fts.fts_server_groups.FTSServerGroups , pass 28 , fail 1 failures so far... fts.fts_server_groups.FTSServerGroups.test_replicas_distribution testrunner logs, diags and results are available under /data/workspace/debian-p0-fts-vset00-00-server-groups_7.0_P1/logs/testrunner-23-Sep-20_21-03-01/test_29 ok ---------------------------------------------------------------------- Ran 1 test in 408.254s