HQ-EvgenyMakarenko:additional_tests evgenymakarenko$ ./testrunner -i ~/projects/pythonedu/cluster.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P1,index_type=scorch,fts_quota=750 -t fts.moving_topology_fts.MovingTopFTS.retry_rebalance_in_during_querying,items=100000,cluster=D,F,num_queries=50,GROUP=P1,retry_time=60,num_retries=1 INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'GROUP': 'P1', 'cluster_name': 'cluster', 'disable_HTP': 'True', 'fts_quota': '750', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'index_type': 'scorch', 'ini': '/Users/evgenymakarenko/projects/pythonedu/cluster.ini', 'num_nodes': 4, 'stop-on-failure': 'False'} INFO:__main__: Number of tests initially selected before GROUP filters: 1 Only cases in GROUPs 'P1' will be executed INFO:__main__:--> Running test: fts.moving_topology_fts.MovingTopFTS.retry_rebalance_in_during_querying,items=100000,cluster=D,F,num_queries=50,GROUP=P1,retry_time=60,num_retries=1 INFO:__main__:Logs folder: /Users/evgenymakarenko/projects/couchbase/master/additional_tests/logs/testrunner-21-Mar-16_15-08-26/test_1 Logs will be stored at /Users/evgenymakarenko/projects/couchbase/master/additional_tests/logs/testrunner-21-Mar-16_15-08-26/test_1 ./testrunner -i /Users/evgenymakarenko/projects/pythonedu/cluster.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P1,index_type=scorch,fts_quota=750 -t fts.moving_topology_fts.MovingTopFTS.retry_rebalance_in_during_querying,items=100000,cluster=D,F,num_queries=50,GROUP=P1,retry_time=60,num_retries=1 Test Input params: {'items': '100000', 'cluster': 'D,F', 'num_queries': '50', 'GROUP': 'P1', 'retry_time': '60', 'num_retries': '1', 'ini': '/Users/evgenymakarenko/projects/pythonedu/cluster.ini', 'cluster_name': 'cluster', 'get-cbcollect-info': 'False', 'disable_HTP': 'True', 'get-logs': 'False', 'stop-on-failure': 'False', 'index_type': 'scorch', 'fts_quota': '750', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/Users/evgenymakarenko/projects/couchbase/master/additional_tests/logs/testrunner-21-Mar-16_15-08-26/test_1'} Run before suite setup for fts.moving_topology_fts.MovingTopFTS.retry_rebalance_in_during_querying -->before_suite_name:fts.moving_topology_fts.MovingTopFTS.suite_setUp,suite: ]> suite_setUp (fts.moving_topology_fts.MovingTopFTS) ... 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [rest_client.set_fts_ram_quota] SUCCESS: FTS RAM quota set to 750mb 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [fts_base.setUp] ==== FTSbasetests setup is started for test #1 suite_setUp ==== 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [rest_client.set_fts_ram_quota] SUCCESS: FTS RAM quota set to 750mb 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] removing nodes from cluster ... 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] cleanup [ip:172.23.107.35 port:8091 ssh_username:root] 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.35, nothing to delete 2021-03-16 15:08:27 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:08:28 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.35:8091 2021-03-16 15:08:28 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:08:28 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.35:8091 is running 2021-03-16 15:08:28 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] Removing user 'cbadminbucket'... 2021-03-16 15:08:28 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.107.35:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-16 15:08:28 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] b'"User was not found."' 2021-03-16 15:08:28 | INFO | MainProcess | MainThread | [fts_base.init_cluster] Initializing Cluster ... 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.35 port:8091 ssh_username:root, nodes/self 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '371', 'memoryTotal': 3961724928, 'memoryFree': 3461648384, 'mcdMemoryReserved': 3022, 'mcdMemoryAllocated': 3022, 'status': 'healthy', 'hostname': '172.23.107.35:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4617-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2024, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.35', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3778, 'curr_items': 0} 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.35:8091:port=8091&username=Administrator&password=password 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-16 15:08:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:08:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:08:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.35: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.35:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.35:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-16 15:08:30 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-16 15:08:31 | INFO | MainProcess | MainThread | [fts_base.init_cluster] 172.23.107.56 will be configured with services fts 2021-03-16 15:08:32 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.56:8091 to cluster 2021-03-16 15:08:32 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.56:8091 to this cluster @172.23.107.35:8091 2021-03-16 15:08:42 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.09 seconds 2021-03-16 15:08:42 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-16 15:08:56 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.35,ns_1@172.23.107.56', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-03-16 15:08:56 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-03-16 15:08:56 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-16 15:08:56 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-03-16 15:09:06 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-03-16 15:09:06 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.229954242706299 sec 2021-03-16 15:09:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:09:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:09:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.35: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-16 15:09:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:09:08 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-16 15:09:08 | INFO | MainProcess | MainThread | [fts_base._enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.107.35 2021-03-16 15:09:08 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.107.35:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-16 15:09:08 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-03-16 15:09:09 | INFO | MainProcess | MainThread | [fts_base._set_bleve_max_result_window] updating bleve_max_result_window of node : ip:172.23.107.56 port:8091 ssh_username:root 2021-03-16 15:09:09 | INFO | MainProcess | MainThread | [rest_client.set_bleve_max_result_window] {"bleveMaxResultWindow": "100000000"} 2021-03-16 15:09:09 | INFO | MainProcess | MainThread | [rest_client.set_bleve_max_result_window] Updated bleveMaxResultWindow 2021-03-16 15:09:09 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.107.35:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1524&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-16 15:09:10 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.07 seconds to create bucket default 2021-03-16 15:09:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:09:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-03-16 15:09:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:09:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-03-16 15:09:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:09:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-03-16 15:09:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:09:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:16 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-03-16 15:09:16 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:09:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-03-16 15:09:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:09:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:09:19 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-03-16 15:09:19 | INFO | MainProcess | MainThread | [fts_base.setUp] ==== FTSbasetests setup is finished for test #1 suite_setUp ==== 2021-03-16 15:09:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:09:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:09:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:09:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:09:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.75 with username:root, attempt#1 of 5 2021-03-16 15:09:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.75 as root 2021-03-16 15:09:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.77 with username:root, attempt#1 of 5 2021-03-16 15:09:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.77 as root 2021-03-16 15:09:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:28 | INFO | MainProcess | MainThread | [moving_topology_fts.suite_setUp] *** MovingTopFTS: suite_setUp() *** 2021-03-16 15:09:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:09:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:09:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:09:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:09:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.75 with username:root, attempt#1 of 5 2021-03-16 15:09:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.75 as root 2021-03-16 15:09:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.77 with username:root, attempt#1 of 5 2021-03-16 15:09:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.77 as root 2021-03-16 15:09:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:09:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:09:37 | INFO | MainProcess | MainThread | [fts_base.tearDown] ==== FTSbasetests cleanup is started for test #1 suite_setUp ==== 2021-03-16 15:09:37 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] removing nodes from cluster ... 2021-03-16 15:09:37 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] cleanup [ip:172.23.107.35 port:8091 ssh_username:root, ip:172.23.107.56 port:8091 ssh_username:root] 2021-03-16 15:09:37 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.107.35 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.107.35 existing buckets : [] 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.107.35 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.35,ns_1@172.23.107.56', 'ejectedNodes': 'ns_1@172.23.107.56', 'user': 'Administrator', 'password': 'password'} 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-03-16 15:09:38 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-16 15:09:48 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2021-03-16 15:10:08 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 30.19 seconds 2021-03-16 15:10:08 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-16 15:10:18 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.107.35 port:8091 ssh_username:root ? [('ns_1@172.23.107.56', 8091)] 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.35:8091 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.35:8091 is running 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.56, nothing to delete 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.56:8091 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.56:8091 is running Cluster instance shutdown with force 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [fts_base.cleanup_cluster] Removing user 'cbadminbucket'... 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [ntonencryptionBase.disable_nton_cluster] Disable up node to node encryption - status = disable and clusterEncryptionLevel = control 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&failoverServerGroup=false 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&failoverServerGroup=false 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&failoverServerGroup=false 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [rest_client.update_autofailover_settings] settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&failoverServerGroup=false 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Changing encryption Level - clusterEncryptionLevel = control 2021-03-16 15:10:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:10:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:10:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:22 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2021-03-16 15:10:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:10:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:10:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:24 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2021-03-16 15:10:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.75 with username:root, attempt#1 of 5 2021-03-16 15:10:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.75 as root 2021-03-16 15:10:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:27 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2021-03-16 15:10:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.77 with username:root, attempt#1 of 5 2021-03-16 15:10:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.77 as root 2021-03-16 15:10:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:29 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli setting-security -c http://localhost -u Administrator -p password --set --cluster-encryption-level control 2021-03-16 15:10:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Output of setting-security command is ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled."] 2021-03-16 15:10:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.change_cluster_encryption_cli] Error of setting-security command is [] 2021-03-16 15:10:29 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Changing node-to-node-encryption to disable 2021-03-16 15:10:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:10:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:10:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:32 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2021-03-16 15:10:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:10:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:10:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:34 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2021-03-16 15:10:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.75 with username:root, attempt#1 of 5 2021-03-16 15:10:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.75 as root 2021-03-16 15:10:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:37 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2021-03-16 15:10:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.77 with username:root, attempt#1 of 5 2021-03-16 15:10:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.77 as root 2021-03-16 15:10:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:39 | INFO | MainProcess | MainThread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --disable 2021-03-16 15:10:40 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Output of node-to-node-encryption command is ['Turned off encryption for node: http://[::1]:8091', 'SUCCESS: Switched node-to-node encryption off'] 2021-03-16 15:10:40 | INFO | MainProcess | MainThread | [ntonencryptionBase.ntonencryption_cli] Error of node-to-node-encryption command is [] 2021-03-16 15:10:40 | INFO | MainProcess | MainThread | [fts_base.tearDown] ==== FTSbasetests cleanup is finished for test #1 suite_setUp === 2021-03-16 15:10:40 | INFO | MainProcess | MainThread | [fts_base.tearDown] closing all ssh connections 2021-03-16 15:10:40 | INFO | MainProcess | MainThread | [fts_base.tearDown] closing all memcached connections Cluster instance shutdown with force ok ---------------------------------------------------------------------- Ran 1 test in 132.809s OK -->result: retry_rebalance_in_during_querying (fts.moving_topology_fts.MovingTopFTS) ... 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [rest_client.set_fts_ram_quota] SUCCESS: FTS RAM quota set to 750mb 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [fts_base.setUp] ==== FTSbasetests setup is started for test #1 retry_rebalance_in_during_querying ==== 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [rest_client.set_fts_ram_quota] SUCCESS: FTS RAM quota set to 750mb 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] removing nodes from cluster ... 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] cleanup [ip:172.23.107.35 port:8091 ssh_username:root] 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.107.35, nothing to delete 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.35:8091 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.35:8091 is running 2021-03-16 15:10:40 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] Removing user 'cbadminbucket'... 2021-03-16 15:10:41 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.107.35:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-16 15:10:41 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] b'"User was not found."' 2021-03-16 15:10:41 | INFO | MainProcess | test_thread | [fts_base.init_cluster] Initializing Cluster ... 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.35 port:8091 ssh_username:root, nodes/self 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '505', 'memoryTotal': 3961724928, 'memoryFree': 3415302144, 'mcdMemoryReserved': 3022, 'mcdMemoryAllocated': 3022, 'status': 'healthy', 'hostname': '172.23.107.35:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4617-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2024, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.35', 'ip': '172.23.107.35', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3778, 'curr_items': 0} 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.35:8091:port=8091&username=Administrator&password=password 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:10:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:10:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.35: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.35:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.35:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-03-16 15:10:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-03-16 15:10:43 | INFO | MainProcess | test_thread | [fts_base.init_cluster] 172.23.107.56 will be configured with services fts 2021-03-16 15:10:44 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.56:8091 to cluster 2021-03-16 15:10:44 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.56:8091 to this cluster @172.23.107.35:8091 2021-03-16 15:10:54 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.10 seconds 2021-03-16 15:10:54 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-16 15:11:07 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.35,ns_1@172.23.107.56', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-03-16 15:11:07 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-03-16 15:11:08 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-16 15:11:08 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-03-16 15:11:18 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-03-16 15:11:18 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.230484962463379 sec 2021-03-16 15:11:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:11:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:11:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:11:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:11:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.35: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [fts_base._enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.107.35 2021-03-16 15:11:20 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.107.35:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [fts_base._set_bleve_max_result_window] updating bleve_max_result_window of node : ip:172.23.107.56 port:8091 ssh_username:root 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [rest_client.set_bleve_max_result_window] {"bleveMaxResultWindow": "100000000"} 2021-03-16 15:11:20 | INFO | MainProcess | test_thread | [rest_client.set_bleve_max_result_window] Updated bleveMaxResultWindow 2021-03-16 15:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.107.35:8091/pools/default/buckets with param: name=default&authType=sasl&saslPassword=None&ramQuotaMB=1524&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-03-16 15:11:21 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.07 seconds to create bucket default 2021-03-16 15:11:21 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:11:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:23 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:23 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-03-16 15:11:23 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:11:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:25 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-03-16 15:11:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:11:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:27 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-03-16 15:11:27 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:11:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:28 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-03-16 15:11:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:11:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:31 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:32 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-03-16 15:11:32 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.107.35 to accept set ops 2021-03-16 15:11:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:11:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-03-16 15:11:34 | INFO | MainProcess | test_thread | [fts_base.setUp] ==== FTSbasetests setup is finished for test #1 retry_rebalance_in_during_querying ==== 2021-03-16 15:11:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:11:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:11:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:11:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:11:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:11:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:11:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:11:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:11:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.75 with username:root, attempt#1 of 5 2021-03-16 15:11:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.75 as root 2021-03-16 15:11:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:11:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:11:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.77 with username:root, attempt#1 of 5 2021-03-16 15:11:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.77 as root 2021-03-16 15:11:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:11:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:11:43 | INFO | MainProcess | test_thread | [fts_base.enable_retry_rebalance] Retry Rebalance settings changed to : {'enabled': True, 'afterTimePeriod': 60, 'maxAttempts': 1} 2021-03-16 15:11:44 | INFO | MainProcess | test_thread | [fts_base.create] Checking if index already exists ... 2021-03-16 15:11:44 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.107.56:8094/api/index/default_index body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_auth: preparePerms, err: index not found b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' auth: Administrator:password 2021-03-16 15:11:44 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.107.56:8094/api/index/default_index body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: rest_auth: preparePerms, err: index not found b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' auth: Administrator:password 2021-03-16 15:11:44 | INFO | MainProcess | test_thread | [fts_base.create] Creating fulltext-index default_index on 172.23.107.56 2021-03-16 15:11:44 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] {"type": "fulltext-index", "name": "default_index", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 0, "maxPartitionsPerPIndex": 171}, "sourceParams": {}} 2021-03-16 15:11:44 | INFO | MainProcess | test_thread | [rest_client.create_fts_index] Index default_index created 2021-03-16 15:11:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validating index distribution for default_index ... 2021-03-16 15:11:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Number of PIndexes = 6 2021-03-16 15:11:45 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Every pIndex serves 171 partitions or lesser 2021-03-16 15:11:45 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Expecting num of partitions in each node in range 853-1024 2021-03-16 15:11:45 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node 7191e49db5319a1ae4a327d156959818 houses 6 pindexes which serve 1024 partitions 2021-03-16 15:11:51 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.107.35:11210 default 2021-03-16 15:13:05 | INFO | MainProcess | test_thread | [fts_base.load_data] Loading phase complete! 2021-03-16 15:13:05 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:06 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 94526, docs in FTS index 'default_index': 57559 2021-03-16 15:13:12 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:12 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100000, docs in FTS index 'default_index': 64436 2021-03-16 15:13:19 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:19 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100000, docs in FTS index 'default_index': 71109 2021-03-16 15:13:26 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:26 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100000, docs in FTS index 'default_index': 79215 2021-03-16 15:13:32 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:33 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100000, docs in FTS index 'default_index': 89146 2021-03-16 15:13:39 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:40 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100000, docs in FTS index 'default_index': 99932 2021-03-16 15:13:46 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.107.35:8091/pools/default/buckets/default/stats?zoom=minute 2021-03-16 15:13:47 | INFO | MainProcess | test_thread | [fts_base.wait_for_indexing_complete] Docs in bucket = 100000, docs in FTS index 'default_index': 100000 2021-03-16 15:13:47 | INFO | MainProcess | Thread-80 | [fts_base.async_rebalance_in] Starting rebalance-in nodes:[ip:172.23.107.77 port:8091 ssh_username:root] at C1 cluster 172.23.107.35 2021-03-16 15:13:48 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.77:8091 to cluster 2021-03-16 15:13:48 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.77:8091 to this cluster @172.23.107.35:8091 2021-03-16 15:13:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:13:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:13:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:13:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:13:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.56: reboot 2021-03-16 15:13:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:13:58 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.09 seconds 2021-03-16 15:13:58 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-03-16 15:14:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.107.35:8091/controller/addNode body: hostname=http%3A%2F%2F172.23.107.77%3A8091&user=Administrator&password=password&services=fts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password 2021-03-16 15:14:22 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.107.35: 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1615932835079, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.107.35' saw that node 'ns_1@172.23.107.56' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2021-03-16T15:13:55.079Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1615932681654, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.107.35\' in 0 seconds.', 'serverTime': '2021-03-16T15:11:21.654Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 12, 'module': 'menelaus_web', 'tstamp': 1615932681555, 'shortText': 'message', 'text': 'Created bucket "default" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,1598029824},\n {durability_min_level,none},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', 'serverTime': '2021-03-16T15:11:21.555Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615932668424, 'shortText': 'message', 'text': 'Rebalance completed successfully.\nRebalance Operation Id = fa691f7631420ea51e5259c779ff5bd5', 'serverTime': '2021-03-16T15:11:08.424Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615932667960, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.107.35','ns_1@172.23.107.56'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = fa691f7631420ea51e5259c779ff5bd5", 'serverTime': '2021-03-16T15:11:07.960Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1615932667844, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-03-16T15:11:07.844Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1615932667692, 'shortText': 'message', 'text': 'Node ns_1@172.23.107.56 joined cluster', 'serverTime': '2021-03-16T15:11:07.692Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1615932667643, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-03-16T15:11:07.643Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1615932667641, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.56\'. Version: "7.0.0-4617-enterprise".', 'serverTime': '2021-03-16T15:11:07.641Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1615932667593, 'shortText': 'message', 'text': 'Conflicting configuration changes to field scramsha_fallback_salt:\n<<10,69,139,189,23,163,129,165,149,145,112,131>> and\n<<23,163,179,52,76,90,147,193,5,24,51,59>>, choosing the former.', 'serverTime': '2021-03-16T15:11:07.593Z'} 2021-03-16 15:14:22 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.107.77: 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1615932835079, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.107.35' saw that node 'ns_1@172.23.107.56' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2021-03-16T15:13:55.079Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1615932681654, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.107.35\' in 0 seconds.', 'serverTime': '2021-03-16T15:11:21.654Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 12, 'module': 'menelaus_web', 'tstamp': 1615932681555, 'shortText': 'message', 'text': 'Created bucket "default" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,1598029824},\n {durability_min_level,none},\n {pitr_enabled,false},\n {pitr_granularity,600},\n {pitr_max_history_age,86400},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', 'serverTime': '2021-03-16T15:11:21.555Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615932668424, 'shortText': 'message', 'text': 'Rebalance completed successfully.\nRebalance Operation Id = fa691f7631420ea51e5259c779ff5bd5', 'serverTime': '2021-03-16T15:11:08.424Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615932667960, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.107.35','ns_1@172.23.107.56'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = fa691f7631420ea51e5259c779ff5bd5", 'serverTime': '2021-03-16T15:11:07.960Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1615932667844, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-03-16T15:11:07.844Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1615932667692, 'shortText': 'message', 'text': 'Node ns_1@172.23.107.56 joined cluster', 'serverTime': '2021-03-16T15:11:07.692Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1615932667643, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-03-16T15:11:07.643Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1615932667641, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.56\'. Version: "7.0.0-4617-enterprise".', 'serverTime': '2021-03-16T15:11:07.641Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1615932667593, 'shortText': 'message', 'text': 'Conflicting configuration changes to field scramsha_fallback_salt:\n<<10,69,139,189,23,163,129,165,149,145,112,131>> and\n<<23,163,179,52,76,90,147,193,5,24,51,59>>, choosing the former.', 'serverTime': '2021-03-16T15:11:07.593Z'} 2021-03-16 15:14:22 | ERROR | MainProcess | Cluster_Thread | [rest_client.add_node] add_node error : b'["Unexpected server error, request logged."]' Traceback (most recent call last): File "lib/tasks/task.py", line 757, in execute self.add_nodes(task_manager) File "lib/tasks/task.py", line 785, in add_nodes node.ip, node.port, services=services_for_node) File "lib/membase/api/rest_client.py", line 1535, in add_node reason=content) membase.api.exception.AddNodeException: Error adding node: 172.23.107.77 to the cluster:172.23.107.35 - b'["Unexpected server error, request logged."]' [, , , , , ] Tue Mar 16 15:14:22 2021 [, , , , , , ] 2021-03-16 15:14:22 | ERROR | MainProcess | Thread-80 | [moving_topology_fts.retry_rebalance] Error adding node: 172.23.107.77 to the cluster:172.23.107.35 - b'["Unexpected server error, request logged."]' 2021-03-16 15:14:22 | INFO | MainProcess | Thread-80 | [fts_base.sleep] sleep for 5 secs. ... 2021-03-16 15:14:27 | INFO | MainProcess | Thread-80 | [fts_base._check_retry_rebalance_succeeded] {'retry_rebalance': 'not_pending'} Exception in thread Thread-80: Traceback (most recent call last): File "pytests/fts/moving_topology_fts.py", line 871, in retry_rebalance services=services) File "pytests/fts/fts_base.py", line 3299, in rebalance_in task.result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception File "lib/tasks/task.py", line 757, in execute self.add_nodes(task_manager) File "lib/tasks/task.py", line 785, in add_nodes node.ip, node.port, services=services_for_node) File "lib/membase/api/rest_client.py", line 1535, in add_node reason=content) membase.api.exception.AddNodeException: Error adding node: 172.23.107.77 to the cluster:172.23.107.35 - b'["Unexpected server error, request logged."]' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "pytests/fts/moving_topology_fts.py", line 878, in retry_rebalance self._check_retry_rebalance_succeeded() File "pytests/fts/fts_base.py", line 3676, in _check_retry_rebalance_succeeded retry_after_secs = result["retry_after_secs"] KeyError: 'retry_after_secs' 2021-03-16 15:14:28 | ERROR | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 172.23.107.56 : [Errno 60] Operation timed out 2021-03-16 15:14:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 10 secs. 2021-03-16 15:14:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#2 of 5 2021-03-16 15:14:38 | ERROR | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] Can't establish SSH session (unknown reason) to node 172.23.107.56 : [Errno None] Unable to connect to port 22 on 172.23.107.56 2021-03-16 15:14:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] Retrying with exponential back off delay for 20 secs. 2021-03-16 15:14:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#3 of 5 2021-03-16 15:15:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:15:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:15:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:15:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.56: iptables -F 2021-03-16 15:15:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:15:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.56: /sbin/iptables -A INPUT -p tcp -i eth0 --dport 1000:65535 -j ACCEPT 2021-03-16 15:15:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:15:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.56: /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.56: /sbin/iptables --list 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Chain INPUT (policy ACCEPT) 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] target prot opt source destination 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] ACCEPT tcp -- anywhere anywhere tcp dpts:cadlock2:65535 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Chain FORWARD (policy ACCEPT) 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] target prot opt source destination 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Chain OUTPUT (policy ACCEPT) 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [remote_util.log_command_output] target prot opt source destination 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.56:8091 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:15:03 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.56:8091 is running 2021-03-16 15:15:03 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 1 ---------------------------------------------------- 2021-03-16 15:15:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:03 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Solita Aarón"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:04 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:04 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Solita Aarón"} is 2802 (took 370.169719ms) 2021-03-16 15:15:04 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 2 ---------------------------------------------------- 2021-03-16 15:15:04 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:04 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Quechua"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:05 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:05 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Quechua"} is 14110 (took 22.046402ms) 2021-03-16 15:15:05 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 3 ---------------------------------------------------- 2021-03-16 15:15:05 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:05 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Bryn Fantine"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Bryn Fantine"} is 9471 (took 16.395856ms) 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 4 ---------------------------------------------------- 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Ciara"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Ciara"} is 1405 (took 1.639744ms) 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 5 ---------------------------------------------------- 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:06 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "bryn@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:10 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:10 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "bryn@mcdiabetes.com"} is 100000 (took 121.040363ms) 2021-03-16 15:15:10 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 6 ---------------------------------------------------- 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Vala Salina"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Vala Salina"} is 9559 (took 11.229917ms) 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 7 ---------------------------------------------------- 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:11 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Sebastián"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Sebastián"} is 1509 (took 2.120242ms) 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 8 ---------------------------------------------------- 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "wiki"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 1.258493ms) 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 9 ---------------------------------------------------- 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:12 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Kacila Araceli"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Kacila Araceli"} is 9594 (took 10.20169ms) 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 10 ---------------------------------------------------- 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Support"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Support"} is 9014 (took 7.976822ms) 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 11 ---------------------------------------------------- 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:13 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Clark"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:14 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:14 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Clark"} is 1443 (took 1.716828ms) 2021-03-16 15:15:14 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 12 ---------------------------------------------------- 2021-03-16 15:15:14 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:14 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Cassandra Vala Kagami"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:15 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:15 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Cassandra Vala Kagami"} is 13743 (took 23.684568ms) 2021-03-16 15:15:15 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 13 ---------------------------------------------------- 2021-03-16 15:15:15 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:15 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "German"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:16 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:16 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "German"} is 14256 (took 13.878036ms) 2021-03-16 15:15:16 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 14 ---------------------------------------------------- 2021-03-16 15:15:16 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:16 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "duvessa@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:21 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:21 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "duvessa@mcdiabetes.com"} is 100000 (took 117.733791ms) 2021-03-16 15:15:21 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 15 ---------------------------------------------------- 2021-03-16 15:15:21 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:21 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Support"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:22 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:22 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Support"} is 9014 (took 8.733774ms) 2021-03-16 15:15:22 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 16 ---------------------------------------------------- 2021-03-16 15:15:22 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:22 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "German"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:23 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:23 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "German"} is 14256 (took 18.905688ms) 2021-03-16 15:15:23 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 17 ---------------------------------------------------- 2021-03-16 15:15:23 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:23 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Engineering"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Engineering"} is 9132 (took 11.451744ms) 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 18 ---------------------------------------------------- 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "wiki"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 0.518732ms) 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 19 ---------------------------------------------------- 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "wiki"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 0.809347ms) 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 20 ---------------------------------------------------- 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:24 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Quechua"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:25 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:25 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Quechua"} is 14110 (took 13.531743ms) 2021-03-16 15:15:25 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 21 ---------------------------------------------------- 2021-03-16 15:15:25 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:25 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "caryssa@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:29 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:29 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "caryssa@mcdiabetes.com"} is 100000 (took 116.853523ms) 2021-03-16 15:15:29 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 22 ---------------------------------------------------- 2021-03-16 15:15:29 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:29 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Marketing"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Marketing"} is 9029 (took 6.891226ms) 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 23 ---------------------------------------------------- 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Maura"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Maura"} is 1354 (took 2.120131ms) 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 24 ---------------------------------------------------- 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:30 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Antonia Bryn Ciara Dominique"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Antonia Bryn Ciara Dominique"} is 17395 (took 20.759088ms) 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 25 ---------------------------------------------------- 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Deirdre Hall"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Deirdre Hall"} is 2882 (took 2.984596ms) 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 26 ---------------------------------------------------- 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:31 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Alvita"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:32 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:32 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Alvita"} is 5087 (took 5.217255ms) 2021-03-16 15:15:32 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 27 ---------------------------------------------------- 2021-03-16 15:15:32 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:32 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Sales"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:33 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:33 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Sales"} is 18127 (took 25.104935ms) 2021-03-16 15:15:33 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 28 ---------------------------------------------------- 2021-03-16 15:15:33 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:33 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "emp"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:38 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:38 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "emp"} is 100000 (took 130.458449ms) 2021-03-16 15:15:38 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 29 ---------------------------------------------------- 2021-03-16 15:15:38 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:38 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "bryn@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:41 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:41 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "bryn@mcdiabetes.com"} is 100000 (took 142.408602ms) 2021-03-16 15:15:42 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 30 ---------------------------------------------------- 2021-03-16 15:15:42 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:42 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Sales"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:43 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:43 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Sales"} is 18127 (took 19.038201ms) 2021-03-16 15:15:43 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 31 ---------------------------------------------------- 2021-03-16 15:15:43 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:43 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Arabic"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Arabic"} is 14325 (took 15.817644ms) 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 32 ---------------------------------------------------- 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Delores"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Delores"} is 1423 (took 2.064708ms) 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 33 ---------------------------------------------------- 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:44 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match": "Safiya Dabria Desdomna"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:45 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:45 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "manages.reports", "match": "Safiya Dabria Desdomna"} is 13954 (took 20.06132ms) 2021-03-16 15:15:45 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 34 ---------------------------------------------------- 2021-03-16 15:15:45 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:45 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Finance"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:46 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:46 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Finance"} is 8873 (took 13.834593ms) 2021-03-16 15:15:46 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 35 ---------------------------------------------------- 2021-03-16 15:15:46 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:46 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "maura@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:50 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:50 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "maura@mcdiabetes.com"} is 100000 (took 103.990949ms) 2021-03-16 15:15:50 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 36 ---------------------------------------------------- 2021-03-16 15:15:50 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:50 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Chinese"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:51 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:51 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Chinese"} is 14134 (took 16.175976ms) 2021-03-16 15:15:51 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 37 ---------------------------------------------------- 2021-03-16 15:15:51 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:51 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Vietnamese"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:52 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:52 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Vietnamese"} is 14336 (took 12.532837ms) 2021-03-16 15:15:52 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 38 ---------------------------------------------------- 2021-03-16 15:15:52 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:52 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "German"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:53 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:53 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "German"} is 14256 (took 13.009044ms) 2021-03-16 15:15:53 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 39 ---------------------------------------------------- 2021-03-16 15:15:53 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:53 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "gallia@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:57 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:57 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "gallia@mcdiabetes.com"} is 100000 (took 158.091022ms) 2021-03-16 15:15:57 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 40 ---------------------------------------------------- 2021-03-16 15:15:57 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:57 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Quechua"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:15:58 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:15:58 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Quechua"} is 14110 (took 12.831358ms) 2021-03-16 15:15:58 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 41 ---------------------------------------------------- 2021-03-16 15:15:58 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:15:58 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "casondrah@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:02 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:02 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "casondrah@mcdiabetes.com"} is 100000 (took 137.123981ms) 2021-03-16 15:16:02 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 42 ---------------------------------------------------- 2021-03-16 15:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:02 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "wiki"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 0.525157ms) 2021-03-16 15:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 43 ---------------------------------------------------- 2021-03-16 15:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:03 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "email", "match": "deirdre@mcdiabetes.com"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:09 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:09 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "email", "match": "deirdre@mcdiabetes.com"} is 100000 (took 157.568086ms) 2021-03-16 15:16:09 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 44 ---------------------------------------------------- 2021-03-16 15:16:09 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:09 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "dept", "match": "Marketing"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "dept", "match": "Marketing"} is 9029 (took 6.739224ms) 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 45 ---------------------------------------------------- 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "wiki"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 0.424166ms) 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 46 ---------------------------------------------------- 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "match": "wiki"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 0.539051ms) 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 47 ---------------------------------------------------- 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:10 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "German"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:11 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:11 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "German"} is 14256 (took 14.431492ms) 2021-03-16 15:16:11 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 48 ---------------------------------------------------- 2021-03-16 15:16:11 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:11 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Malay"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Malay"} is 14320 (took 12.368948ms) 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 49 ---------------------------------------------------- 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Kacila Phillips"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "name", "match": "Kacila Phillips"} is 2763 (took 3.40206ms) 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [task.execute] ----------------------------------------------------- Query # 50 ---------------------------------------------------- 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:12 | INFO | MainProcess | Cluster_Thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"field": "languages_known", "match": "Africans"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:14 | INFO | MainProcess | Cluster_Thread | [task.execute] Status: {'total': 6, 'failed': 0, 'successful': 6} 2021-03-16 15:16:14 | INFO | MainProcess | Cluster_Thread | [task.execute] FTS hits for query: {"field": "languages_known", "match": "Africans"} is 14224 (took 12.760864ms) 2021-03-16 15:16:14 | INFO | MainProcess | test_thread | [moving_topology_fts.run_tasks_and_report] SUCCESS: 50 out of 50 queries passed 2021-03-16 15:16:14 | INFO | MainProcess | test_thread | [fts_base.sleep] sleep for 30 secs. ... 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validating index distribution for default_index ... 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Number of PIndexes = 6 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Every pIndex serves 171 partitions or lesser 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Expecting num of partitions in each node in range 853-1024 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [fts_base.is_index_partitioned_balanced] Validated: Node 7191e49db5319a1ae4a327d156959818 houses 6 pindexes which serve 1024 partitions 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:44 | INFO | MainProcess | test_thread | [fts_base.run_fts_query] Running query {"indexName": "default_index", "size": 10000000, "from": 0, "explain": false, "query": {"match": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node: 172.23.107.56:8094 2021-03-16 15:16:49 | INFO | MainProcess | test_thread | [fts_base.execute_query] SUCCESS! Expected hits: 100000, fts returned: 100000 2021-03-16 15:16:49 | INFO | MainProcess | test_thread | [moving_topology_fts.retry_rebalance_in_during_querying] SUCCESS! Hits: 100000 2021-03-16 15:16:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.35 with username:root, attempt#1 of 5 2021-03-16 15:16:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.35 as root 2021-03-16 15:16:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:16:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:16:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:16:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:16:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:16:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:16:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.75 with username:root, attempt#1 of 5 2021-03-16 15:16:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.75 as root 2021-03-16 15:16:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:16:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:16:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.77 with username:root, attempt#1 of 5 2021-03-16 15:16:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.77 as root 2021-03-16 15:16:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:16:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:16:59 | INFO | MainProcess | test_thread | [fts_base.tearDown] ==== FTSbasetests cleanup is started for test #1 retry_rebalance_in_during_querying ==== 2021-03-16 15:16:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:16:59 | INFO | MainProcess | test_thread | [fts_base.delete] Deleting fulltext-index default_index on 172.23.107.56 2021-03-16 15:17:00 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:17:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.56 with username:root, attempt#1 of 5 2021-03-16 15:17:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.56 as root 2021-03-16 15:17:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-16 15:17:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-16 15:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.56: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep ^default_index | wc -l 2021-03-16 15:17:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-03-16 15:17:01 | INFO | MainProcess | test_thread | [fts_base.are_index_files_deleted_from_disk] 0 2021-03-16 15:17:03 | INFO | MainProcess | test_thread | [fts_base.delete] Validated: all index files for default_index deleted from disk 2021-03-16 15:17:03 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] removing nodes from cluster ... 2021-03-16 15:17:03 | INFO | MainProcess | test_thread | [fts_base.cleanup_cluster] cleanup [ip:172.23.107.35 port:8091 ssh_username:root, ip:172.23.107.56 port:8091 ssh_username:root, ip:172.23.107.77 port:8091 ssh_username:root] 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.107.35 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.107.35 existing buckets : [] 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.107.35 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-03-16 15:17:04 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.35,ns_1@172.23.107.56,ns_1@172.23.107.77', 'ejectedNodes': 'ns_1@172.23.107.56,ns_1@172.23.107.77', 'user': 'Administrator', 'password': 'password'} 2021-03-16 15:17:05 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2021-03-16 15:17:05 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-16 15:17:15 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-16 15:17:25 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-03-16 15:17:35 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.107.35: 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615933055024, 'shortText': 'message', 'text': "Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.107.77',{error,timeout}}]}}}.\nRebalance Operation Id = 2a363d2f3a4492f65e0dcc99c0a40baf", 'serverTime': '2021-03-16T15:17:35.024Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1615933025021, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.107.35'], EjectNodes = ['ns_1@172.23.107.56',\n 'ns_1@172.23.107.77'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 2a363d2f3a4492f65e0dcc99c0a40baf", 'serverTime': '2021-03-16T15:17:05.021Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1615933024977, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@172.23.107.35\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\nStarting rebalance from test, ejected nodes [\'ns_1@172.23.107.56\', \'ns_1@172.23.107.77\']', 'serverTime': '2021-03-16T15:17:04.977Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 11, 'module': 'menelaus_web', 'tstamp': 1615933024655, 'shortText': 'message', 'text': 'Deleted bucket "default"\n', 'serverTime': '2021-03-16T15:17:04.655Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1615933024101, 'shortText': 'message', 'text': 'Shutting down bucket "default" on \'ns_1@172.23.107.35\' for deletion', 'serverTime': '2021-03-16T15:17:04.101Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1615932898099, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.56\'. Version: "7.0.0-4617-enterprise".', 'serverTime': '2021-03-16T15:14:58.099Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1615932897584, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.107.35' saw that node 'ns_1@172.23.107.56' came up. Tags: []", 'serverTime': '2021-03-16T15:14:57.584Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1615932897577, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.107.56' saw that node 'ns_1@172.23.107.35' came up. Tags: []", 'serverTime': '2021-03-16T15:14:57.577Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.56', 'type': 'info', 'code': 2, 'module': 'ns_cookie_manager', 'tstamp': 1615932897545, 'shortText': 'cookie update', 'text': 'Node \'ns_1@172.23.107.56\' synchronized otp cookie {sanitized,\n <<"kh9CnWP47Kka4ppGmaM3BjBAJbug2lmLq23zwHPGK2I=">>} from cluster', 'serverTime': '2021-03-16T15:14:57.545Z'} 2021-03-16 15:17:35 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.35', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1615932835079, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.107.35' saw that node 'ns_1@172.23.107.56' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2021-03-16T15:13:55.079Z'} Cluster instance shutdown with force 2021-03-16 15:17:35 | INFO | MainProcess | test_thread | [fts_base.tearDown] closing all ssh connections 2021-03-16 15:17:35 | INFO | MainProcess | test_thread | [fts_base.tearDown] closing all memcached connections Cluster instance shutdown with force ERROR ====================================================================== ERROR: retry_rebalance_in_during_querying (fts.moving_topology_fts.MovingTopFTS) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/fts/moving_topology_fts.py", line 20, in tearDown super(MovingTopFTS, self).tearDown() File "pytests/fts/fts_base.py", line 3786, in tearDown self._cb_cluster.cleanup_cluster(self) File "pytests/fts/fts_base.py", line 2272, in cleanup_cluster ClusterOperationHelper.cleanup_cluster([node]) File "lib/membase/helper/cluster_helper.py", line 267, in cleanup_cluster wait_for_rebalance=wait_for_rebalance) File "lib/membase/api/rest_client.py", line 116, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1790, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1889, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1866, in _rebalance_status_and_progress raise RebalanceFailedException(msg) membase.api.exception.RebalanceFailedException: Rebalance Failed: {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 415.306s FAILED (errors=1) downloading 172.23.107.35 ..............................................downloading 172.23.107.56 ............................................downloading 172.23.107.75 ........................................downloading 172.23.107.77 ........................................summary so far suite fts.moving_topology_fts.MovingTopFTS , pass 0 , fail 1 failures so far... fts.moving_topology_fts.MovingTopFTS.retry_rebalance_in_during_querying testrunner logs, diags and results are available under /Users/evgenymakarenko/projects/couchbase/master/additional_tests/logs/testrunner-21-Mar-16_15-08-26/test_1 *** Tests executed count: 1 Run after suite setup for fts.moving_topology_fts.MovingTopFTS.retry_rebalance_in_during_querying suite_tearDown (fts.moving_topology_fts.MovingTopFTS) ... 2021-03-16 15:17:42 | INFO | MainProcess | MainThread | [rest_client.get_nodes] Node 172.23.107.77 not part of cluster inactiveAdded 2021-03-16 15:17:57 | ERROR | MainProcess | MainThread | [rest_client._http_request] POST http://172.23.107.35:8091/pools/default body: ftsMemoryQuota=750 headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password ERROR ====================================================================== ERROR: suite_tearDown (fts.moving_topology_fts.MovingTopFTS) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/fts/moving_topology_fts.py", line 17, in setUp super(MovingTopFTS, self).setUp() File "pytests/fts/fts_base.py", line 3616, in setUp sdk_compression=sdk_compression) File "pytests/fts/fts_base.py", line 2009, in __init__ self.__set_fts_ram_quota() File "pytests/fts/fts_base.py", line 2019, in __set_fts_ram_quota RestConnection(self.__master_node).set_fts_ram_quota(fts_quota) File "lib/membase/api/rest_client.py", line 3253, in set_fts_ram_quota raise Exception("Error setting fts ram quota: {0}".format(content)) Exception: Error setting fts ram quota: b'["Unexpected server error, request logged."]' ---------------------------------------------------------------------- Ran 1 test in 15.264s FAILED (errors=1) During the test, Remote Connections: 0, Disconnections: 76 ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... HQ-EvgenyMakarenko:additional_tests evgenymakarenko$