[global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 eventing_port:8096 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [elastic] ip:172.23.107.206 port:9200 [_1] ip:172.23.104.98 [_2] ip:172.23.105.191 [_3] ip:172.23.105.123 [_4] ip:172.23.105.44 [_5] ip:172.23.100.26 [_6] ip:172.23.104.202 [_7] ip:172.23.104.211 [_8] ip:172.23.104.234 + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... + cd testrunner + git checkout neo Switched to a new branch 'neo' Branch neo set up to track remote branch neo from origin. + git submodule init Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet' Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' + git submodule update --init --force --remote Cloning into 'gauntlet'... Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300' Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out 'de89b059ce28567dbac18afb032271a4eaa674ff' + cp ../../node.ini . + cp ../../debug.conf . + python3 testrunner.py -i node.ini -c debug.conf -p upgrade_version=7.2.0-5325,stop-on-failure=False,skip_cleanup=False,skip_init_check_cbserver=true,groups=simple,gsi_type=memory_optimized,upgrade_test=True,dgm_run=True,skip_log_scan=true,upgrade_version=7.2.0-5325 INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: debug.conf INFO:__main__:Test prefix: newupgradetests.MultiNodesUpgradeTests INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: newupgradetests.MultiNodesUpgradeTests.test_offline_upgrade_with_add_new_services,initial_version=6.5.1-6296,nodes_init=4,ddocs_num=3,pre_upgrade=create_replica_index,doc-per-day=1,dataset=default,after_upgrade_buckets_flush=True,groups=simple,initial-services-setting=kv-kv,n1ql,index-kv,index,after_upgrade_services_in=index,num_index_replica=2,disable_HTP=True,offline_failover_upgrade=True,upgrade_test=True,dgm_run=True,init_nodes=False INFO:__main__:Logs folder: /data/workspace/temp_centos7_4node_jython/tr_repo_dir/testrunner/logs/testrunner-23-May-20_06-53-34/test_1 *** TestRunner *** {'cluster_name': 'node', 'conf_file': 'debug.conf', 'dgm_run': 'True', 'groups': 'simple', 'gsi_type': 'memory_optimized', 'ini': 'node.ini', 'num_nodes': 8, 'skip_cleanup': 'False', 'skip_init_check_cbserver': 'true', 'skip_log_scan': 'true', 'spec': 'debug', 'stop-on-failure': 'False', 'upgrade_test': 'True', 'upgrade_version': '7.2.0-5325'} Logs will be stored at /data/workspace/temp_centos7_4node_jython/tr_repo_dir/testrunner/logs/testrunner-23-May-20_06-53-34/test_1 ./testrunner -i node.ini -p upgrade_version=7.2.0-5325,stop-on-failure=False,skip_cleanup=False,skip_init_check_cbserver=true,groups=simple,gsi_type=memory_optimized,upgrade_test=True,dgm_run=True,skip_log_scan=true,upgrade_version=7.2.0-5325 -t newupgradetests.MultiNodesUpgradeTests.test_offline_upgrade_with_add_new_services,initial_version=6.5.1-6296,nodes_init=4,ddocs_num=3,pre_upgrade=create_replica_index,doc-per-day=1,dataset=default,after_upgrade_buckets_flush=True,groups=simple,initial-services-setting=kv-kv,n1ql,index-kv,index,after_upgrade_services_in=index,num_index_replica=2,disable_HTP=True,offline_failover_upgrade=True,upgrade_test=True,dgm_run=True,init_nodes=False Test Input params: {'initial_version': '6.5.1-6296', 'nodes_init': '4', 'ddocs_num': '3', 'pre_upgrade': 'create_replica_index', 'doc-per-day': '1', 'dataset': 'default', 'after_upgrade_buckets_flush': 'True', 'groups': 'simple', 'initial-services-setting': 'kv-kv,n1ql,index-kv,index', 'after_upgrade_services_in': 'index', 'num_index_replica': '2', 'disable_HTP': 'True', 'offline_failover_upgrade': 'True', 'upgrade_test': 'True', 'dgm_run': 'True', 'init_nodes': 'False', 'ini': 'node.ini', 'cluster_name': 'node', 'spec': 'debug', 'conf_file': 'debug.conf', 'upgrade_version': '7.2.0-5325', 'stop-on-failure': 'False', 'skip_cleanup': 'False', 'skip_init_check_cbserver': 'true', 'gsi_type': 'memory_optimized', 'skip_log_scan': 'true', 'num_nodes': 8, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/temp_centos7_4node_jython/tr_repo_dir/testrunner/logs/testrunner-23-May-20_06-53-34/test_1'} lib/couchbase_helper/data_analysis_helper.py:905: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" ./pytests/security/x509main.py:80: DeprecationWarning: invalid escape sequence \. ''' ./pytests/security/x509main.py:90: DeprecationWarning: invalid escape sequence \ def _generate_cert(self, servers, root_cn='Root\ Authority', type='go', encryption="", key_length=1024, client_ip=None, alt_names='default', dns=None, uri=None,wildcard_dns=None): lib/couchbase_helper/tuq_generators.py:113: DeprecationWarning: invalid escape sequence \w regex = re.compile("[\w']+\.[\w']+") lib/couchbase_helper/tuq_generators.py:363: DeprecationWarning: invalid escape sequence \[ diff = set(order_clause.split(',')) - set(re.compile('doc\["[\w\']+"\]').findall(select_clause)) /data/workspace/temp_centos7_4node_jython/tr_repo_dir/testrunner/scripts/install.py:1072: DeprecationWarning: invalid escape sequence \d if re.match('\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', server.ip): pytests/security/SecretsMasterBase.py:63: DeprecationWarning: invalid escape sequence \. status, ipAddress = subprocess.getstatusoutput("ifconfig eth0 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | awk '{print $2}'") test_offline_upgrade_with_add_new_services (newupgradetests.MultiNodesUpgradeTests) ... 2023-05-20 06:53:35,589 - root - INFO - ============== EventingBaseTest setup has started ============== 2023-05-20 06:53:35,590 - root - INFO - Starting Test: test_offline_upgrade_with_add_new_services test run as below. Change params as needed. newupgradetests.MultiNodesUpgradeTests.test_offline_upgrade_with_add_new_services, initial_version=4.x.x-xxx,nodes_init=2,ddocs_num=3,pre_upgrade=create_index, post_upgrade=drop_index,doc-per-day=1,dataset=default,groups=simple, after_upgrade_buckets_flush=True,skip_init_check_cbserver=True, gsi_type=memory_optimized,init_nodes=False,upgrade_version=5.5.0-xxx, initial-services-setting='kv,index-kv,n1ql',after_upgrade_services_in=eventing, dgm_run=True,upgrade_test=True if test failover upgrade, add param offline_failover_upgrade=True 2023-05-20 06:53:35,592 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:35,757 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:36,421 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:36,871 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:36,873 - root - INFO - ============== basetestcase setup was started for test #1 test_offline_upgrade_with_add_new_services============== 2023-05-20 06:53:36,874 - root - INFO - any cluster operation in setup will be skipped 2023-05-20 06:53:36,874 - root - INFO - ============== basetestcase setup was finished for test #1 test_offline_upgrade_with_add_new_services ============== 2023-05-20 06:53:36,876 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:37,045 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:37,677 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:38,133 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:38,596 - root - INFO - ============== EventingBaseTest setup has completed ============== 2023-05-20 06:53:38,598 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:38,743 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:39,235 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:39,683 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:40,416 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:40,560 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:41,130 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:41,560 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:41,561 - root - INFO - will install 6.5.1-6296 on ['172.23.104.98', '172.23.105.191', '172.23.105.123', '172.23.105.44'] 2023-05-20 06:53:41,564 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:41,567 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:53:41,568 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:53:41,570 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:53:41,680 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:53:41,705 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:53:41,726 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:41,730 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:53:41,905 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:41,975 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:42,290 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:42,291 - root - INFO - 172.23.105.44 *****In couchbase uninstall**** 2023-05-20 06:53:42,291 - root - INFO - CentOS 2023-05-20 06:53:42,336 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:42,345 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:42,346 - root - INFO - 172.23.105.123 *****In couchbase uninstall**** 2023-05-20 06:53:42,346 - root - INFO - CentOS 2023-05-20 06:53:42,402 - root - INFO - running command.raw on 172.23.105.44: killall -9 rpm 2023-05-20 06:53:42,406 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:42,429 - root - INFO - couchbase server at 172.23.105.123 may not installed yet 2023-05-20 06:53:42,429 - root - INFO - running command.raw on 172.23.105.123: killall -9 rpm 2023-05-20 06:53:42,437 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2023-05-20 06:53:42,438 - root - ERROR - rpm: no process found 2023-05-20 06:53:42,438 - root - INFO - running command.raw on 172.23.105.44: rm -f /var/lib/rpm/.rpm.lock 2023-05-20 06:53:42,459 - root - INFO - command executed with root but got an error ['bash: killall: command not found'] ... 2023-05-20 06:53:42,460 - root - ERROR - bash: killall: command not found 2023-05-20 06:53:42,460 - root - INFO - running command.raw on 172.23.105.123: rm -f /var/lib/rpm/.rpm.lock 2023-05-20 06:53:42,492 - root - INFO - command executed successfully with root 2023-05-20 06:53:42,533 - root - INFO - command executed successfully with root 2023-05-20 06:53:42,533 - root - INFO - running rpm -e to remove couchbase-server 2023-05-20 06:53:42,534 - root - INFO - running command.raw on 172.23.105.123: rpm -e couchbase-server 2023-05-20 06:53:42,600 - root - INFO - running rpm -e to remove couchbase-server 2023-05-20 06:53:42,601 - root - INFO - running command.raw on 172.23.105.44: rpm -e couchbase-server 2023-05-20 06:53:42,658 - root - INFO - command executed with root but got an error ['error: package couchbase-server is not installed'] ... 2023-05-20 06:53:42,658 - root - INFO - running command.raw on 172.23.105.123: pkill -u couchbase 2023-05-20 06:53:42,695 - root - INFO - command executed successfully with root 2023-05-20 06:53:42,696 - root - INFO - running command.raw on 172.23.105.123: ps -ef | grep couchbase 2023-05-20 06:53:42,797 - root - INFO - command executed successfully with root 2023-05-20 06:53:42,826 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:42,827 - root - INFO - 172.23.104.98 *****In couchbase uninstall**** 2023-05-20 06:53:42,827 - root - INFO - CentOS 2023-05-20 06:53:42,906 - root - INFO - couchbase server at 172.23.104.98 may not installed yet 2023-05-20 06:53:42,906 - root - INFO - running command.raw on 172.23.104.98: killall -9 rpm 2023-05-20 06:53:42,974 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2023-05-20 06:53:42,975 - root - ERROR - rpm: no process found 2023-05-20 06:53:42,975 - root - INFO - running command.raw on 172.23.104.98: rm -f /var/lib/rpm/.rpm.lock 2023-05-20 06:53:43,010 - root - INFO - command executed successfully with root 2023-05-20 06:53:43,010 - root - INFO - running rpm -e to remove couchbase-server 2023-05-20 06:53:43,011 - root - INFO - running command.raw on 172.23.104.98: rpm -e couchbase-server 2023-05-20 06:53:43,058 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:43,059 - root - INFO - 172.23.105.191 *****In couchbase uninstall**** 2023-05-20 06:53:43,059 - root - INFO - CentOS 2023-05-20 06:53:43,167 - root - INFO - couchbase server at 172.23.105.191 may not installed yet 2023-05-20 06:53:43,167 - root - INFO - running command.raw on 172.23.105.191: killall -9 rpm 2023-05-20 06:53:43,177 - root - INFO - command executed with root but got an error ['error: package couchbase-server is not installed'] ... 2023-05-20 06:53:43,178 - root - INFO - running command.raw on 172.23.104.98: pkill -u couchbase 2023-05-20 06:53:43,218 - root - INFO - command executed successfully with root 2023-05-20 06:53:43,218 - root - INFO - running command.raw on 172.23.104.98: ps -ef | grep couchbase 2023-05-20 06:53:43,275 - root - INFO - command executed with root but got an error ['rpm: no process found'] ... 2023-05-20 06:53:43,275 - root - ERROR - rpm: no process found 2023-05-20 06:53:43,275 - root - INFO - running command.raw on 172.23.105.191: rm -f /var/lib/rpm/.rpm.lock 2023-05-20 06:53:43,315 - root - INFO - command executed successfully with root 2023-05-20 06:53:43,387 - root - INFO - command executed successfully with root 2023-05-20 06:53:43,387 - root - INFO - running rpm -e to remove couchbase-server 2023-05-20 06:53:43,387 - root - INFO - running command.raw on 172.23.105.191: rpm -e couchbase-server 2023-05-20 06:53:43,544 - root - INFO - command executed with root but got an error ['error: package couchbase-server is not installed'] ... 2023-05-20 06:53:43,545 - root - INFO - running command.raw on 172.23.105.191: pkill -u couchbase 2023-05-20 06:53:43,638 - root - INFO - command executed successfully with root 2023-05-20 06:53:43,638 - root - INFO - running command.raw on 172.23.105.191: ps -ef | grep couchbase 2023-05-20 06:53:43,735 - root - INFO - command executed successfully with root 2023-05-20 06:53:44,450 - root - INFO - running command.raw on 172.23.105.123: ipcrm 2023-05-20 06:53:44,524 - root - INFO - command executed successfully with root 2023-05-20 06:53:44,952 - root - INFO - running command.raw on 172.23.104.98: ipcrm 2023-05-20 06:53:45,023 - root - INFO - command executed successfully with root thread uninstaller-thread-172.23.104.98 finished 2023-05-20 06:53:46,151 - root - INFO - running command.raw on 172.23.105.191: ipcrm 2023-05-20 06:53:46,268 - root - INFO - command executed successfully with root thread uninstaller-thread-172.23.105.191 finished thread uninstaller-thread-172.23.105.123 finished 2023-05-20 06:53:47,213 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave', 'warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couch ... 2023-05-20 06:53:47,214 - root - INFO - running command.raw on 172.23.105.44: pkill -u couchbase 2023-05-20 06:53:47,238 - root - INFO - command executed successfully with root 2023-05-20 06:53:47,238 - root - INFO - running command.raw on 172.23.105.44: ps -ef | grep couchbase 2023-05-20 06:53:47,311 - root - INFO - command executed successfully with root 2023-05-20 06:53:48,739 - root - INFO - running command.raw on 172.23.105.44: ipcrm 2023-05-20 06:53:48,794 - root - INFO - command executed successfully with root thread uninstaller-thread-172.23.105.44 finished 2023-05-20 06:53:48,797 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:48,968 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:49,572 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:50,035 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:50,119 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:53:50,273 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:53:50,886 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:51,532 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:51,614 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:53:51,767 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:53:51,968 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:52,412 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:52,498 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:53:52,606 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:53:52,899 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:53,212 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:53,269 - root - INFO - ********CouchbaseServerInstaller:install 2023-05-20 06:53:53,270 - root - INFO - ********CouchbaseServerInstaller:install 2023-05-20 06:53:53,272 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:53:53,272 - root - INFO - ********CouchbaseServerInstaller:install 2023-05-20 06:53:53,274 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:53,275 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:53:53,276 - root - INFO - ********CouchbaseServerInstaller:install 2023-05-20 06:53:53,280 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:53:53,382 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:53:53,406 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:53:53,428 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:53:53,434 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:53,606 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:53,673 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:53,990 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.105.44 is centos 7 *** 2023-05-20 06:53:53,993 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:53:54,007 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:54,032 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:54,054 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.105.123 is centos 7 *** 2023-05-20 06:53:54,057 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:53:54,096 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:53:54,185 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:53:54,388 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:54,393 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:54,443 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.104.98 is centos 7 *** 2023-05-20 06:53:54,446 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:54,614 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:54,681 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 --> remote server architecture_type..x86_64 *** OS version of this server 172.23.105.191 is centos 7 *** 2023-05-20 06:53:54,684 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:53:54,707 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:54,708 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:53:54,841 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:53:54,842 - root - INFO - SSH Connected to 172.23.105.191 as root check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:54,842 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:53:54,902 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:54,903 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:54,906 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:53:54,909 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:53:55,015 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:53:55,039 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:53:55,233 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:55,237 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:55,323 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:55,509 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:55,649 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm name : couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm product : product_version : 6.5.1-6296 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2023-05-20 06:53:55,650 - root - INFO - running command.raw on 172.23.105.44: /sbin/iptables -F 2023-05-20 06:53:55,683 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm name : couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm product : product_version : 6.5.1-6296 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2023-05-20 06:53:55,683 - root - INFO - running command.raw on 172.23.105.123: /sbin/iptables -F 2023-05-20 06:53:55,690 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:55,690 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:53:55,694 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:55,696 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:55,710 - root - INFO - command executed successfully with root 2023-05-20 06:53:55,711 - root - INFO - running command.raw on 172.23.105.44: /sbin/iptables -t nat -F 2023-05-20 06:53:55,725 - root - INFO - command executed successfully with root 2023-05-20 06:53:55,726 - root - INFO - running command.raw on 172.23.105.123: /sbin/iptables -t nat -F 2023-05-20 06:53:55,768 - root - INFO - command executed successfully with root 2023-05-20 06:53:55,768 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:53:55,800 - root - INFO - command executed successfully with root 2023-05-20 06:53:55,800 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:53:55,861 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:55,874 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:53:55,926 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:53:56,183 - root - INFO - running command.raw on 172.23.105.123: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 06:53:56,235 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 check if http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:56,236 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:53:56,238 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm is live 2023-05-20 06:53:56,240 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:53:56,288 - root - INFO - running command.raw on 172.23.105.44: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 06:53:56,401 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:53:56,468 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:56,908 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm name : couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm product : product_version : 6.5.1-6296 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2023-05-20 06:53:56,909 - root - INFO - running command.raw on 172.23.104.98: /sbin/iptables -F 2023-05-20 06:53:56,946 - root - INFO - command executed successfully with root 2023-05-20 06:53:56,947 - root - INFO - running command.raw on 172.23.104.98: /sbin/iptables -t nat -F 2023-05-20 06:53:57,021 - root - INFO - command executed successfully with root 2023-05-20 06:53:57,021 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:53:57,062 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:53:57,181 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:53:57,810 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Downloading the build...url : http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm name : couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm product : product_version : 6.5.1-6296 os : deliverable_type : rpm architecture_type : x86_64 toy : toy- 2023-05-20 06:53:57,811 - root - INFO - running command.raw on 172.23.105.191: /sbin/iptables -F 2023-05-20 06:53:57,905 - root - INFO - command executed successfully with root 2023-05-20 06:53:57,905 - root - INFO - running command.raw on 172.23.105.191: /sbin/iptables -t nat -F 2023-05-20 06:53:57,923 - root - INFO - running command.raw on 172.23.104.98: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 06:53:57,993 - root - INFO - command executed successfully with root 2023-05-20 06:53:57,993 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:53:58,170 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:53:58,911 - root - INFO - running command.raw on 172.23.105.191: cd /tmp;wget -nc -q -O couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/mad-hatter/6296/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 06:53:59,015 - root - INFO - command executed successfully with root 2023-05-20 06:53:59,016 - root - INFO - total 337M 2023-05-20 06:53:59,293 - root - INFO - -rw-r--r--. 1 root root 337M Oct 18 2022 couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:53:59,293 - root - INFO - drwxr-xr-x. 3 couchbase couchbase 20 May 19 19:04 data 2023-05-20 06:53:59,293 - root - INFO - drwxr-xr-x. 4 couchbase couchbase 35 May 19 18:57 index 2023-05-20 06:53:59,294 - root - INFO - drwx------. 3 root root 16 Mar 28 08:17 systemd-private-4490784fc3cf419ea9ec0ce02ea4eb49-ntpd.service-QDU0x1 2023-05-20 06:53:59,384 - root - INFO - *****install server *** 2023-05-20 06:53:59,385 - root - INFO - -->deliverable_type : rpm 2023-05-20 06:53:59,385 - root - INFO - /tmp/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm or /tmp/ 2023-05-20 06:53:59,477 - root - INFO - --> execute_command: end... --> inside rpm check... 2023-05-20 06:53:59,478 - root - INFO - running command.raw on 172.23.105.191: cat /etc/redhat-release 2023-05-20 06:53:59,604 - root - INFO - command executed successfully with root 2023-05-20 06:53:59,605 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2023-05-20 06:54:01,374 - root - INFO - command executed successfully with root 2023-05-20 06:54:01,374 - root - INFO - total 337M 2023-05-20 06:54:01,375 - root - INFO - -rw-r--r--. 1 root root 337M Oct 18 2022 couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:54:01,375 - root - INFO - drwx------. 3 root root 16 Mar 13 09:01 systemd-private-356447aa087e462d9c6fbb42a35324e1-ntpd.service-43UyhJ 2023-05-20 06:54:01,390 - root - INFO - command executed successfully with root 2023-05-20 06:54:01,391 - root - INFO - total 337M 2023-05-20 06:54:01,391 - root - INFO - -rw-r--r-- 1 root root 337M Oct 18 2022 couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:54:01,391 - root - INFO - drwxrwxrwx 2 root root 6 May 20 05:30 sg_logs 2023-05-20 06:54:01,391 - root - INFO - drwx------ 3 root root 16 Apr 19 07:49 systemd-private-3ebe73983dad4ffe83894d045f6584e6-ntpd.service-hcBhNh 2023-05-20 06:54:01,393 - root - INFO - command executed successfully with root 2023-05-20 06:54:01,394 - root - INFO - total 337M 2023-05-20 06:54:01,394 - root - INFO - -rw-r--r--. 1 root root 337M Oct 18 2022 couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm 2023-05-20 06:54:01,394 - root - INFO - drwx------. 3 root root 17 May 19 11:00 systemd-private-a21e5daf5f9f4891bbed337ba41a1ca3-ntpd.service-8BDkU9 2023-05-20 06:54:01,438 - root - INFO - *****install server *** 2023-05-20 06:54:01,439 - root - INFO - -->deliverable_type : rpm 2023-05-20 06:54:01,439 - root - INFO - /tmp/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm or /tmp/ 2023-05-20 06:54:01,441 - root - INFO - *****install server *** 2023-05-20 06:54:01,441 - root - INFO - -->deliverable_type : rpm 2023-05-20 06:54:01,441 - root - INFO - /tmp/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm or /tmp/ 2023-05-20 06:54:01,445 - root - INFO - *****install server *** 2023-05-20 06:54:01,446 - root - INFO - -->deliverable_type : rpm 2023-05-20 06:54:01,446 - root - INFO - /tmp/couchbase-server-enterprise-6.5.1-6296-centos7.x86_64.rpm or /tmp/ 2023-05-20 06:54:01,507 - root - INFO - --> execute_command: end... --> inside rpm check... 2023-05-20 06:54:01,507 - root - INFO - running command.raw on 172.23.105.44: cat /etc/redhat-release 2023-05-20 06:54:01,519 - root - INFO - --> execute_command: end... --> inside rpm check... 2023-05-20 06:54:01,519 - root - INFO - running command.raw on 172.23.105.123: cat /etc/redhat-release 2023-05-20 06:54:01,521 - root - INFO - --> execute_command: end... --> inside rpm check... 2023-05-20 06:54:01,521 - root - INFO - running command.raw on 172.23.104.98: cat /etc/redhat-release 2023-05-20 06:54:01,563 - root - INFO - command executed successfully with root 2023-05-20 06:54:01,564 - root - INFO - CentOS Linux release 7.9.2009 (Core) 2023-05-20 06:54:01,594 - root - INFO - command executed successfully with root 2023-05-20 06:54:01,595 - root - INFO - CentOS Linux release 7.4.1708 (Core) 2023-05-20 06:54:01,596 - root - INFO - command executed successfully with root 2023-05-20 06:54:01,596 - root - INFO - CentOS Linux release 7.6.1810 (Core) 2023-05-20 06:54:53,380 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2023-05-20 06:54:53,380 - root - INFO - Warming: RPMDB altered outside of yum 2023-05-20 06:54:53,443 - root - INFO - Checking if the directory /tmp exists or not. 2023-05-20 06:54:53,444 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2023-05-20 06:54:53,524 - root - INFO - running command.raw on 172.23.105.44: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 06:54:53,581 - root - INFO - command executed successfully with root 2023-05-20 06:54:53,581 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:54:53,663 - root - INFO - Starting couchbase server 2023-05-20 06:54:53,772 - root - INFO - Running systemd command on this server 2023-05-20 06:54:53,772 - root - INFO - running command.raw on 172.23.105.44: systemctl start couchbase-server.service 2023-05-20 06:54:53,795 - root - INFO - command executed successfully with root 2023-05-20 06:54:53,796 - root - INFO - 172.23.105.44:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 06:54:53,865 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2023-05-20 06:54:53,866 - root - INFO - Warming: RPMDB altered outside of yum 2023-05-20 06:54:53,908 - root - INFO - Checking if the directory /tmp exists or not. 2023-05-20 06:54:53,910 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2023-05-20 06:54:54,037 - root - INFO - running command.raw on 172.23.105.123: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 06:54:54,114 - root - INFO - command executed successfully with root 2023-05-20 06:54:54,115 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:54:54,213 - root - INFO - Starting couchbase server 2023-05-20 06:54:54,396 - root - INFO - Running systemd command on this server 2023-05-20 06:54:54,396 - root - INFO - running command.raw on 172.23.105.123: systemctl start couchbase-server.service 2023-05-20 06:54:54,442 - root - INFO - command executed successfully with root 2023-05-20 06:54:54,443 - root - INFO - 172.23.105.123:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 06:54:58,801 - root - INFO - running command.raw on 172.23.105.44: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 06:54:58,831 - root - INFO - command executed successfully with root 2023-05-20 06:54:58,831 - root - INFO - Couchbase server status: [] 2023-05-20 06:54:58,832 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:54:58,906 - root - INFO - process beam.smp is running on 172.23.105.44: with pid 128408 2023-05-20 06:54:58,959 - root - INFO - wait 30 seconds for Couchbase server to start 2023-05-20 06:54:59,448 - root - INFO - running command.raw on 172.23.105.123: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 06:54:59,483 - root - INFO - command executed successfully with root 2023-05-20 06:54:59,483 - root - INFO - Couchbase server status: [] 2023-05-20 06:54:59,483 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:54:59,580 - root - INFO - process beam.smp is running on 172.23.105.123: with pid 124034 2023-05-20 06:54:59,652 - root - INFO - wait 30 seconds for Couchbase server to start 2023-05-20 06:55:39,778 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2023-05-20 06:55:39,778 - root - INFO - Warming: RPMDB altered outside of yum 2023-05-20 06:55:39,824 - root - INFO - Checking if the directory /tmp exists or not. 2023-05-20 06:55:39,825 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2023-05-20 06:55:39,917 - root - INFO - running command.raw on 172.23.104.98: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 06:55:39,999 - root - INFO - command executed successfully with root 2023-05-20 06:55:40,000 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:55:40,116 - root - INFO - Starting couchbase server 2023-05-20 06:55:40,286 - root - INFO - Running systemd command on this server 2023-05-20 06:55:40,287 - root - INFO - running command.raw on 172.23.104.98: systemctl start couchbase-server.service 2023-05-20 06:55:40,413 - root - INFO - command executed successfully with root 2023-05-20 06:55:40,413 - root - INFO - 172.23.104.98:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 06:55:44,521 - root - INFO - If couchbase server is running with this error. Go to log_command_output to add error mesg to bypass it. 2023-05-20 06:55:44,522 - root - INFO - Warming: RPMDB altered outside of yum 2023-05-20 06:55:44,623 - root - INFO - Checking if the directory /tmp exists or not. 2023-05-20 06:55:44,625 - root - ERROR - Directory at /tmp DOES exist. Fx returns True 2023-05-20 06:55:44,800 - root - INFO - running command.raw on 172.23.105.191: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 06:55:44,918 - root - INFO - command executed successfully with root 2023-05-20 06:55:44,919 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:55:45,083 - root - INFO - Starting couchbase server 2023-05-20 06:55:45,339 - root - INFO - Running systemd command on this server 2023-05-20 06:55:45,339 - root - INFO - running command.raw on 172.23.105.191: systemctl start couchbase-server.service 2023-05-20 06:55:45,414 - root - INFO - running command.raw on 172.23.104.98: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 06:55:45,463 - root - INFO - command executed successfully with root 2023-05-20 06:55:45,463 - root - INFO - Couchbase server status: [] 2023-05-20 06:55:45,464 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:55:45,522 - root - INFO - process beam.smp is running on 172.23.104.98: with pid 1635 2023-05-20 06:55:45,527 - root - INFO - command executed successfully with root 2023-05-20 06:55:45,527 - root - INFO - 172.23.105.191:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 06:55:45,594 - root - INFO - wait 30 seconds for Couchbase server to start 2023-05-20 06:55:50,533 - root - INFO - running command.raw on 172.23.105.191: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 06:55:50,635 - root - INFO - command executed successfully with root 2023-05-20 06:55:50,636 - root - INFO - Couchbase server status: [] 2023-05-20 06:55:50,636 - root - INFO - Checking for process beam.smp on linux 2023-05-20 06:55:50,773 - root - INFO - process beam.smp is running on 172.23.105.191: with pid 19887 2023-05-20 06:55:50,876 - root - INFO - wait 30 seconds for Couchbase server to start thread installer-thread-172.23.104.98 finished thread installer-thread-172.23.105.191 finished thread installer-thread-172.23.105.123 finished thread installer-thread-172.23.105.44 finished 2023-05-20 06:56:20,909 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:56:20,911 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:56:20,914 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:56:20,918 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:56:21,029 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:56:21,044 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:56:21,063 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:56:21,087 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:56:21,236 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:21,329 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:21,641 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:21,645 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:21,673 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:21,704 - root - INFO - 172.23.105.44 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:21,704 - root - INFO - creating RestConnection(ip:172.23.105.44 port:8091 ssh_username:root) and the server type is 2023-05-20 06:56:21,713 - root - INFO - Done creating rest connection 2023-05-20 06:56:21,761 - root - INFO - 172.23.105.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:21,761 - root - INFO - creating RestConnection(ip:172.23.105.123 port:8091 ssh_username:root) and the server type is 2023-05-20 06:56:21,772 - root - INFO - Done creating rest connection 2023-05-20 06:56:21,852 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:22,083 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:22,168 - root - INFO - 172.23.104.98 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:22,168 - root - INFO - creating RestConnection(ip:172.23.104.98 port:8091 ssh_username:root) and the server type is 2023-05-20 06:56:22,182 - root - INFO - Done creating rest connection 2023-05-20 06:56:22,566 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:22,660 - root - INFO - 172.23.105.191 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:22,661 - root - INFO - creating RestConnection(ip:172.23.105.191 port:8091 ssh_username:root) and the server type is 2023-05-20 06:56:22,672 - root - INFO - Done creating rest connection 2023-05-20 06:56:24,721 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:24,729 - root - INFO - Node version in cluster 6.5.1-6296-enterprise 2023-05-20 06:56:24,730 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2023-05-20 06:56:24,782 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:24,798 - root - INFO - Node version in cluster 6.5.1-6296-enterprise 2023-05-20 06:56:24,798 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2023-05-20 06:56:24,969 - root - INFO - check if man installed on vm? 2023-05-20 06:56:24,970 - root - INFO - running command.raw on 172.23.105.44: rpm -qa | grep 'man-db' 2023-05-20 06:56:25,116 - root - INFO - check if man installed on vm? 2023-05-20 06:56:25,117 - root - INFO - running command.raw on 172.23.105.123: rpm -qa | grep 'man-db' 2023-05-20 06:56:25,192 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:25,208 - root - INFO - Node version in cluster 6.5.1-6296-enterprise 2023-05-20 06:56:25,208 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2023-05-20 06:56:25,526 - root - INFO - check if man installed on vm? 2023-05-20 06:56:25,527 - root - INFO - running command.raw on 172.23.104.98: rpm -qa | grep 'man-db' 2023-05-20 06:56:25,682 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:25,694 - root - INFO - Node version in cluster 6.5.1-6296-enterprise 2023-05-20 06:56:25,694 - root - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** 2023-05-20 06:56:26,115 - root - INFO - check if man installed on vm? 2023-05-20 06:56:26,115 - root - INFO - running command.raw on 172.23.105.191: rpm -qa | grep 'man-db' 2023-05-20 06:56:26,210 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,210 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.105.44 2023-05-20 06:56:26,211 - root - INFO - running command.raw on 172.23.105.44: unzip 2023-05-20 06:56:26,267 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,268 - root - INFO - unzip command is ready 2023-05-20 06:56:26,268 - root - INFO - Check if ntp is installed 2023-05-20 06:56:26,268 - root - INFO - This OS version centos 7 2023-05-20 06:56:26,268 - root - INFO - running command.raw on 172.23.105.44: systemctl status ntpd 2023-05-20 06:56:26,335 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,335 - root - INFO - running command.raw on 172.23.105.44: ntpstat 2023-05-20 06:56:26,397 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,397 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.105.123 2023-05-20 06:56:26,397 - root - INFO - running command.raw on 172.23.105.123: unzip 2023-05-20 06:56:26,429 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,429 - root - INFO - running command.raw on 172.23.105.44: timedatectl status 2023-05-20 06:56:26,436 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,436 - root - INFO - unzip command is ready 2023-05-20 06:56:26,436 - root - INFO - Check if ntp is installed 2023-05-20 06:56:26,436 - root - INFO - This OS version centos 7 2023-05-20 06:56:26,436 - root - INFO - running command.raw on 172.23.105.123: systemctl status ntpd 2023-05-20 06:56:26,512 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,513 - root - INFO - running command.raw on 172.23.105.44: date 2023-05-20 06:56:26,524 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,524 - root - INFO - running command.raw on 172.23.105.123: ntpstat 2023-05-20 06:56:26,568 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,680 - root - INFO - ['Sat May 20 06:56:26 PDT 2023'] IP: 172.23.105.44 2023-05-20 06:56:26,698 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,698 - root - INFO - running command.raw on 172.23.105.123: timedatectl status 2023-05-20 06:56:26,812 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,813 - root - INFO - running command.raw on 172.23.105.123: date 2023-05-20 06:56:26,849 - root - INFO - command executed successfully with root 2023-05-20 06:56:26,999 - root - INFO - ['Sat May 20 06:56:26 PDT 2023'] IP: 172.23.105.123 2023-05-20 06:56:27,110 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,110 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.104.98 2023-05-20 06:56:27,111 - root - INFO - running command.raw on 172.23.104.98: unzip 2023-05-20 06:56:27,188 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,188 - root - INFO - unzip command is ready 2023-05-20 06:56:27,188 - root - INFO - Check if ntp is installed 2023-05-20 06:56:27,188 - root - INFO - This OS version centos 7 2023-05-20 06:56:27,189 - root - INFO - running command.raw on 172.23.104.98: systemctl status ntpd 2023-05-20 06:56:27,284 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,285 - root - INFO - running command.raw on 172.23.104.98: ntpstat 2023-05-20 06:56:27,435 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,436 - root - INFO - running command.raw on 172.23.104.98: timedatectl status 2023-05-20 06:56:27,562 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,562 - root - INFO - running command.raw on 172.23.104.98: date 2023-05-20 06:56:27,633 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,783 - root - INFO - ['Sat May 20 06:56:27 PDT 2023'] IP: 172.23.104.98 thread initializer-thread-172.23.104.98 finished 2023-05-20 06:56:27,894 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,895 - root - INFO - ---> Run command unzip to check if it is ready on server 172.23.105.191 2023-05-20 06:56:27,895 - root - INFO - running command.raw on 172.23.105.191: unzip 2023-05-20 06:56:27,968 - root - INFO - command executed successfully with root 2023-05-20 06:56:27,968 - root - INFO - unzip command is ready 2023-05-20 06:56:27,968 - root - INFO - Check if ntp is installed 2023-05-20 06:56:27,968 - root - INFO - This OS version centos 7 2023-05-20 06:56:27,969 - root - INFO - running command.raw on 172.23.105.191: systemctl status ntpd 2023-05-20 06:56:28,100 - root - INFO - command executed successfully with root 2023-05-20 06:56:28,100 - root - INFO - running command.raw on 172.23.105.191: ntpstat 2023-05-20 06:56:28,241 - root - INFO - command executed successfully with root 2023-05-20 06:56:28,241 - root - INFO - running command.raw on 172.23.105.191: timedatectl status 2023-05-20 06:56:28,410 - root - INFO - command executed successfully with root 2023-05-20 06:56:28,410 - root - INFO - running command.raw on 172.23.105.191: date 2023-05-20 06:56:28,525 - root - INFO - command executed successfully with root 2023-05-20 06:56:28,679 - root - INFO - ['Sat May 20 06:56:28 PDT 2023'] IP: 172.23.105.191 thread initializer-thread-172.23.105.191 finished thread initializer-thread-172.23.105.123 finished thread initializer-thread-172.23.105.44 finished 2023-05-20 06:56:28,682 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:56:28,855 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:56:29,430 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:29,853 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:29,854 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.98 **** 2023-05-20 06:56:29,862 - root - ERROR - GET http://172.23.104.98:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-05-20 06:56:29,868 - root - INFO - At least one of the node in the cluster is on pre-spock version. Not creating user since RBAC is a spock feature. 2023-05-20 06:56:29,868 - root - INFO - sleep for 5 secs. ... 2023-05-20 06:56:34,873 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2023-05-20 06:56:34,883 - root - ERROR - GET http://172.23.104.98:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-05-20 06:56:34,885 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:56:35,045 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:56:35,595 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:36,025 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:36,068 - root - INFO - 172.23.104.98 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:36,068 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2023-05-20 06:56:41,079 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:41,086 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:56:41,252 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:56:41,921 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:42,523 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:42,610 - root - INFO - 172.23.105.191 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:42,610 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2023-05-20 06:56:47,622 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:47,629 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:56:47,759 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:56:47,955 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:48,400 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:48,489 - root - INFO - 172.23.105.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:48,490 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2023-05-20 06:56:53,498 - root - INFO - -->is_ns_server_running? 2023-05-20 06:56:53,506 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:56:53,615 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:56:53,921 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:56:54,237 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:56:54,301 - root - INFO - 172.23.105.44 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 06:56:54,301 - root - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... 2023-05-20 06:56:59,310 - root - INFO - -->is_ns_server_running? 2023-05-20 06:57:00,109 - root - INFO - server: ip:172.23.104.98 port:8091 ssh_username:root, nodes/self 2023-05-20 06:57:00,113 - root - INFO - {'uptime': '76', 'memoryTotal': 3963314176, 'memoryFree': 3443879936, 'mcdMemoryReserved': 3023, 'mcdMemoryAllocated': 3023, 'status': 'healthy', 'hostname': '172.23.104.98:8091', 'clusterCompatibility': 393221, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.5.1-6296-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1601, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.98', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3779, 'curr_items': 0} 2023-05-20 06:57:00,116 - root - ERROR - GET http://172.23.104.98:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-05-20 06:57:00,118 - root - INFO - pools/default params : memoryQuota=2025 2023-05-20 06:57:00,127 - root - INFO - --> init_node_services(Administrator,password,172.23.104.98,8091,['kv']) 2023-05-20 06:57:00,127 - root - INFO - /node/controller/setupServices params on 172.23.104.98: 8091:hostname=172.23.104.98&user=Administrator&password=password&services=kv 2023-05-20 06:57:00,139 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 06:57:00,140 - root - INFO - settings/web params on 172.23.104.98:8091:port=8091&username=Administrator&password=password 2023-05-20 06:57:00,234 - root - INFO - --> status:True 2023-05-20 06:57:00,238 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 06:57:00,408 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 06:57:00,827 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:57:01,258 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:57:01,259 - root - INFO - running command.raw on 172.23.104.98: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-05-20 06:57:01,316 - root - INFO - command executed successfully with root 2023-05-20 06:57:01,317 - root - INFO - ['ok'] 2023-05-20 06:57:01,321 - root - INFO - /diag/eval status on 172.23.104.98:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2023-05-20 06:57:01,321 - root - INFO - settings/indexes params : storageMode=memory_optimized 2023-05-20 06:57:01,352 - root - INFO - server: ip:172.23.105.191 port:8091 ssh_username:root, nodes/self 2023-05-20 06:57:01,357 - root - INFO - {'uptime': '71', 'memoryTotal': 3963314176, 'memoryFree': 3498364928, 'mcdMemoryReserved': 3023, 'mcdMemoryAllocated': 3023, 'status': 'healthy', 'hostname': '172.23.105.191:8091', 'clusterCompatibility': 393221, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.5.1-6296-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1601, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.191', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3779, 'curr_items': 0} 2023-05-20 06:57:01,360 - root - ERROR - GET http://172.23.105.191:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-05-20 06:57:01,360 - root - INFO - pools/default params : memoryQuota=2025 2023-05-20 06:57:01,371 - root - INFO - --> init_node_services(Administrator,password,172.23.105.191,8091,['kv,n1ql,index']) 2023-05-20 06:57:01,372 - root - INFO - /node/controller/setupServices params on 172.23.105.191: 8091:hostname=172.23.105.191&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex 2023-05-20 06:57:01,386 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 06:57:01,386 - root - INFO - settings/web params on 172.23.105.191:8091:port=8091&username=Administrator&password=password 2023-05-20 06:57:01,478 - root - INFO - --> status:True 2023-05-20 06:57:01,480 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 06:57:01,637 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 06:57:02,191 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:57:02,766 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:57:02,766 - root - INFO - running command.raw on 172.23.105.191: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-05-20 06:57:02,851 - root - INFO - command executed successfully with root 2023-05-20 06:57:02,852 - root - INFO - ['ok'] 2023-05-20 06:57:02,856 - root - INFO - /diag/eval status on 172.23.105.191:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2023-05-20 06:57:02,856 - root - INFO - settings/indexes params : storageMode=memory_optimized 2023-05-20 06:57:02,889 - root - INFO - server: ip:172.23.105.123 port:8091 ssh_username:root, nodes/self 2023-05-20 06:57:02,894 - root - INFO - {'uptime': '124', 'memoryTotal': 4156542976, 'memoryFree': 3555426304, 'mcdMemoryReserved': 3171, 'mcdMemoryAllocated': 3171, 'status': 'healthy', 'hostname': '172.23.105.123:8091', 'clusterCompatibility': 393221, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.5.1-6296-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1712, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.123', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3963, 'curr_items': 0} 2023-05-20 06:57:02,897 - root - ERROR - GET http://172.23.105.123:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-05-20 06:57:02,897 - root - INFO - pools/default params : memoryQuota=2124 2023-05-20 06:57:02,908 - root - INFO - --> init_node_services(Administrator,password,172.23.105.123,8091,['kv,index']) 2023-05-20 06:57:02,908 - root - INFO - /node/controller/setupServices params on 172.23.105.123: 8091:hostname=172.23.105.123&user=Administrator&password=password&services=kv%2Cindex 2023-05-20 06:57:02,920 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 06:57:02,920 - root - INFO - settings/web params on 172.23.105.123:8091:port=8091&username=Administrator&password=password 2023-05-20 06:57:02,973 - root - INFO - --> status:True 2023-05-20 06:57:02,975 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 06:57:03,095 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 06:57:03,291 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:57:03,729 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:57:03,730 - root - INFO - running command.raw on 172.23.105.123: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-05-20 06:57:03,826 - root - INFO - command executed successfully with root 2023-05-20 06:57:03,826 - root - INFO - ['ok'] 2023-05-20 06:57:03,831 - root - INFO - /diag/eval status on 172.23.105.123:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2023-05-20 06:57:03,831 - root - INFO - settings/indexes params : storageMode=memory_optimized 2023-05-20 06:57:03,868 - root - INFO - server: ip:172.23.105.44 port:8091 ssh_username:root, nodes/self 2023-05-20 06:57:03,871 - root - INFO - {'uptime': '128', 'memoryTotal': 4201582592, 'memoryFree': 3667361792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.44:8091', 'clusterCompatibility': 393221, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.5.1-6296-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.44', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} 2023-05-20 06:57:03,874 - root - ERROR - GET http://172.23.105.44:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2023-05-20 06:57:03,874 - root - INFO - pools/default params : memoryQuota=2147 2023-05-20 06:57:03,879 - root - INFO - --> init_node_services(Administrator,password,172.23.105.44,8091,['kv,fts,index']) 2023-05-20 06:57:03,880 - root - INFO - /node/controller/setupServices params on 172.23.105.44: 8091:hostname=172.23.105.44&user=Administrator&password=password&services=kv%2Cfts%2Cindex 2023-05-20 06:57:03,887 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 06:57:03,887 - root - INFO - settings/web params on 172.23.105.44:8091:port=8091&username=Administrator&password=password 2023-05-20 06:57:03,930 - root - INFO - --> status:True 2023-05-20 06:57:03,932 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 06:57:04,043 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 06:57:04,355 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 06:57:04,671 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 06:57:04,672 - root - INFO - running command.raw on 172.23.105.44: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2023-05-20 06:57:04,740 - root - INFO - command executed successfully with root 2023-05-20 06:57:04,740 - root - INFO - ['ok'] 2023-05-20 06:57:04,743 - root - INFO - /diag/eval status on 172.23.105.44:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2023-05-20 06:57:04,743 - root - INFO - settings/indexes params : storageMode=memory_optimized 2023-05-20 06:57:04,758 - root - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.98 **** 2023-05-20 06:57:04,778 - root - ERROR - DELETE http://172.23.104.98:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2023-05-20 06:57:04,779 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2023-05-20 06:57:04,886 - root - INFO - sleep for 5 secs. ... 2023-05-20 06:57:09,891 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2023-05-20 06:57:09,910 - root - INFO - sleep for 20 secs. wait to make sure node is ready ... 2023-05-20 06:57:30,787 - root - INFO - adding node 172.23.105.191:8091 to cluster 2023-05-20 06:57:30,800 - root - INFO - adding remote node @172.23.105.191:18091 to this cluster @172.23.104.98:8091 2023-05-20 06:57:40,819 - root - INFO - rebalance progress took 10.02 seconds 2023-05-20 06:57:40,819 - root - INFO - sleep for 10 seconds after rebalance... 2023-05-20 06:57:56,113 - root - INFO - Node 172.23.105.191 not part of cluster inactiveAdded 2023-05-20 06:57:56,129 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.191', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 06:57:56,140 - root - INFO - rebalance operation started 2023-05-20 06:58:06,155 - root - INFO - Rebalance - status: none, progress: 100.00% 2023-05-20 06:58:06,162 - root - INFO - rebalancing was completed with progress: 100% in 10.022544145584106 sec 2023-05-20 06:58:06,163 - root - INFO - sleep for 10 secs. ... 2023-05-20 06:58:17,175 - root - INFO - adding node 172.23.105.123:8091 to cluster 2023-05-20 06:58:17,192 - root - INFO - adding remote node @172.23.105.123:18091 to this cluster @172.23.104.98:8091 2023-05-20 06:58:27,206 - root - INFO - rebalance progress took 10.01 seconds 2023-05-20 06:58:27,206 - root - INFO - sleep for 10 seconds after rebalance... 2023-05-20 06:58:40,414 - root - INFO - Node 172.23.105.123 not part of cluster inactiveAdded 2023-05-20 06:58:40,425 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.123,ns_1@172.23.105.191', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 06:58:40,432 - root - INFO - rebalance operation started 2023-05-20 06:58:50,446 - root - INFO - Rebalance - status: none, progress: 100.00% 2023-05-20 06:58:50,456 - root - INFO - rebalancing was completed with progress: 100% in 10.024027824401855 sec 2023-05-20 06:58:50,456 - root - INFO - sleep for 10 secs. ... 2023-05-20 06:59:01,468 - root - INFO - adding node 172.23.105.44:8091 to cluster 2023-05-20 06:59:01,485 - root - INFO - adding remote node @172.23.105.44:18091 to this cluster @172.23.104.98:8091 2023-05-20 06:59:11,503 - root - INFO - rebalance progress took 10.02 seconds 2023-05-20 06:59:11,503 - root - INFO - sleep for 10 seconds after rebalance... 2023-05-20 06:59:23,951 - root - INFO - Node 172.23.105.44 not part of cluster inactiveAdded 2023-05-20 06:59:23,965 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.123,ns_1@172.23.105.191,ns_1@172.23.105.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 06:59:23,972 - root - INFO - rebalance operation started 2023-05-20 06:59:33,986 - root - INFO - Rebalance - status: none, progress: 100.00% 2023-05-20 06:59:33,997 - root - INFO - rebalancing was completed with progress: 100% in 10.024945259094238 sec 2023-05-20 06:59:33,997 - root - INFO - sleep for 10 secs. ... 2023-05-20 06:59:45,085 - root - INFO - http://172.23.104.98:8091/pools/default/buckets with param: name=default&ramQuotaMB=256&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2023-05-20 06:59:45,094 - root - INFO - 0.01 seconds to create bucket default 2023-05-20 06:59:45,094 - root - INFO - waiting for memcached bucket : default in 172.23.104.98 to accept set ops 2023-05-20 06:59:45,788 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 06:59:45,876 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 06:59:46,033 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 06:59:46,140 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 06:59:46,323 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 06:59:46,443 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 06:59:46,535 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 06:59:46,684 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 06:59:47,951 - root - INFO - bucket 'default' was created with per node RAM quota: 256 2023-05-20 06:59:48,006 - root - ERROR - http://172.23.107.206:9200/es_index error 404 reason: status: 404, content: b'{"error":"IndexMissingException[[es_index] missing]","status":404}' b'{"error":"IndexMissingException[[es_index] missing]","status":404}' 2023-05-20 06:59:48,009 - root - ERROR - http://172.23.107.206:9200/es_index error 404 reason: status: 404, content: b'{"error":"IndexMissingException[[es_index] missing]","status":404}' b'{"error":"IndexMissingException[[es_index] missing]","status":404}' 2023-05-20 06:59:48,073 - root - INFO - Created empty index 'es_index' on Elastic Search node with custom standard analyzer(default) 2023-05-20 06:59:48,121 - root - INFO - Checking if index already exists ... 2023-05-20 06:59:48,122 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=GET params= client_cert=None verify=False 2023-05-20 06:59:48,129 - root - ERROR - b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' 2023-05-20 06:59:48,129 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=DELETE params= client_cert=None verify=False 2023-05-20 06:59:48,135 - root - ERROR - b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n' 2023-05-20 06:59:48,135 - root - INFO - Creating fulltext-index index_default on 172.23.105.44 2023-05-20 06:59:48,135 - root - INFO - {"type": "fulltext-index", "name": "index_default", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {}, "sourceParams": {}} 2023-05-20 06:59:48,136 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=PUT params={"type": "fulltext-index", "name": "index_default", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {}, "sourceParams": {}} client_cert=None verify=False 2023-05-20 06:59:48,170 - root - INFO - Index index_default created 2023-05-20 06:59:50,200 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=GET params= client_cert=None verify=False 2023-05-20 06:59:50,299 - root - INFO - Starting operation 'create' on Elastic Search ... 2023-05-20 06:59:50,464 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 06:59:50,576 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 06:59:50,700 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 06:59:50,868 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 06:59:51,695 - root - INFO - 1000 documents bulk loaded into ES 2023-05-20 06:59:51,990 - root - INFO - ES index count for 'es_index': 1000 2023-05-20 06:59:52,021 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/count verb=GET params= client_cert=None verify=False 2023-05-20 06:59:52,036 - root - INFO - http://172.23.104.98:8091/pools/default/buckets/default/stats?zoom=minute 2023-05-20 06:59:52,061 - root - INFO - Docs in bucket = 0, docs in FTS index 'index_default': 1000, docs in ES index: 1000 2023-05-20 06:59:58,095 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/count verb=GET params= client_cert=None verify=False 2023-05-20 06:59:58,108 - root - INFO - http://172.23.104.98:8091/pools/default/buckets/default/stats?zoom=minute 2023-05-20 06:59:58,141 - root - INFO - Docs in bucket = 1000, docs in FTS index 'index_default': 1000, docs in ES index: 1000 2023-05-20 06:59:58,982 - root - INFO - ----------------------------------------------------- Query # 1 ---------------------------------------------------- 2023-05-20 06:59:59,001 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"disjuncts": [{"field": "name", "match": "Solita Aarón"}, {"field": "languages_known", "match": "Urdu"}, {"field": "languages_known", "match": "Quechua"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,001 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"disjuncts": [{"field": "name", "match": "Solita Aar\xc3\xb3n"}, {"field": "languages_known", "match": "Urdu"}, {"field": "languages_known", "match": "Quechua"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,024 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,024 - root - INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Solita Aarón"}, {"field": "languages_known", "match": "Urdu"}, {"field": "languages_known", "match": "Quechua"}]} is 266 (took 3.383623ms) 2023-05-20 06:59:59,025 - root - INFO - ES query '{'query': {'bool': {'should': [{'match': {'name': 'Solita Aarón'}}, {'match': {'languages_known': 'Urdu'}}, {'match': {'languages_known': 'Quechua'}}]}}}' 2023-05-20 06:59:59,155 - root - INFO - ES hits for query: {"query": {"bool": {"should": [{"match": {"name": "Solita Aarón"}}, {"match": {"languages_known": "Urdu"}}, {"match": {"languages_known": "Quechua"}}]}}} on es_index is 266 (took 107ms) 2023-05-20 06:59:59,156 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,156 - root - INFO - ----------------------------------------------------- Query # 2 ---------------------------------------------------- 2023-05-20 06:59:59,181 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,181 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,188 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,189 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 145 (took 0.796165ms) 2023-05-20 06:59:59,189 - root - INFO - ES query '{'query': {'query_string': {'query': '+languages_known:"Malay"'}}}' 2023-05-20 06:59:59,224 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "+languages_known:\"Malay\""}}} on es_index is 145 (took 24ms) 2023-05-20 06:59:59,225 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,225 - root - INFO - ----------------------------------------------------- Query # 3 ---------------------------------------------------- 2023-05-20 06:59:59,240 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "email", "match": "ciara@mcdiabetes.com"}, {"field": "manages.reports", "match": "Balandria"}, {"field": "type", "match": "wiki"}]}, "should": {"disjuncts": [{"field": "manages.reports", "match": "Joby Desdomna Vala Salina"}, {"field": "languages_known", "match": "Malay"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,240 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "email", "match": "ciara@mcdiabetes.com"}, {"field": "manages.reports", "match": "Balandria"}, {"field": "type", "match": "wiki"}]}, "should": {"disjuncts": [{"field": "manages.reports", "match": "Joby Desdomna Vala Salina"}, {"field": "languages_known", "match": "Malay"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,248 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,249 - root - INFO - FTS hits for query: {"must": {"conjuncts": [{"field": "email", "match": "ciara@mcdiabetes.com"}, {"field": "manages.reports", "match": "Balandria"}, {"field": "type", "match": "wiki"}]}, "should": {"disjuncts": [{"field": "manages.reports", "match": "Joby Desdomna Vala Salina"}, {"field": "languages_known", "match": "Malay"}]}} is 0 (took 0.584736ms) 2023-05-20 06:59:59,249 - root - INFO - ES query '{'query': {'bool': {'must': [{'match': {'email': 'ciara@mcdiabetes.com'}}, {'match': {'manages.reports': 'Balandria'}}, {'match': {'type': 'wiki'}}], 'should': [{'match': {'manages.reports': 'Joby Desdomna Vala Salina'}}, {'match': {'languages_known': 'Malay'}}]}}}' 2023-05-20 06:59:59,256 - root - INFO - ES hits for query: {"query": {"bool": {"must": [{"match": {"email": "ciara@mcdiabetes.com"}}, {"match": {"manages.reports": "Balandria"}}, {"match": {"type": "wiki"}}], "should": [{"match": {"manages.reports": "Joby Desdomna Vala Salina"}}, {"match": {"languages_known": "Malay"}}]}}} on es_index is 0 (took 4ms) 2023-05-20 06:59:59,256 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,257 - root - INFO - ----------------------------------------------------- Query # 4 ---------------------------------------------------- 2023-05-20 06:59:59,271 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "ilith Smith", "fuzziness": 0}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,271 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "ilith Smith", "fuzziness": 0}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,279 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,279 - root - INFO - FTS hits for query: {"field": "name", "match": "ilith Smith", "fuzziness": 0} is 0 (took 3.034572ms) 2023-05-20 06:59:59,279 - root - INFO - ES query '{'query': {'fuzzy': {'name': {'value': 'ilith Smith', 'fuzziness': 0}}}}' 2023-05-20 06:59:59,293 - root - INFO - ES hits for query: {"query": {"fuzzy": {"name": {"value": "ilith Smith", "fuzziness": 0}}}} on es_index is 0 (took 10ms) 2023-05-20 06:59:59,293 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,293 - root - INFO - ----------------------------------------------------- Query # 5 ---------------------------------------------------- 2023-05-20 06:59:59,309 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "wildcard": "e*"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,309 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "wildcard": "e*"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,320 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,320 - root - INFO - FTS hits for query: {"field": "type", "wildcard": "e*"} is 1000 (took 2.316804ms) 2023-05-20 06:59:59,320 - root - INFO - ES query '{'query': {'wildcard': {'type': 'e*'}}}' 2023-05-20 06:59:59,436 - root - INFO - ES hits for query: {"query": {"wildcard": {"type": "e*"}}} on es_index is 1000 (took 79ms) 2023-05-20 06:59:59,437 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,437 - root - INFO - ----------------------------------------------------- Query # 6 ---------------------------------------------------- 2023-05-20 06:59:59,455 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1974-10-08T07:51:00", "end": "2023-05-20T06:59:58.167441", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,455 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1974-10-08T07:51:00", "end": "2023-05-20T06:59:58.167441", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,464 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,464 - root - INFO - FTS hits for query: {"field": "join_date", "start": "1974-10-08T07:51:00", "end": "2023-05-20T06:59:58.167441", "inclusive_start": true, "inclusive_end": true} is 654 (took 1.131173ms) 2023-05-20 06:59:59,464 - root - INFO - ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gte': '1974-10-08T07:51:00', 'lte': '2023-05-20T06:59:58.167441'}}}}}}' 2023-05-20 06:59:59,510 - root - INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gte": "1974-10-08T07:51:00", "lte": "2023-05-20T06:59:58.167441"}}}}}} on es_index is 654 (took 29ms) 2023-05-20 06:59:59,510 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,511 - root - INFO - ----------------------------------------------------- Query # 7 ---------------------------------------------------- 2023-05-20 06:59:59,528 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match_phrase": "Basha"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,528 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match_phrase": "Basha"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,534 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,534 - root - INFO - FTS hits for query: {"field": "manages.reports", "match_phrase": "Basha"} is 37 (took 0.433692ms) 2023-05-20 06:59:59,534 - root - INFO - ES query '{'query': {'match_phrase': {'manages.reports': 'Basha'}}}' 2023-05-20 06:59:59,542 - root - INFO - ES hits for query: {"query": {"match_phrase": {"manages.reports": "Basha"}}} on es_index is 37 (took 3ms) 2023-05-20 06:59:59,542 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,543 - root - INFO - ----------------------------------------------------- Query # 8 ---------------------------------------------------- 2023-05-20 06:59:59,559 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"conjuncts": [{"field": "name", "match": "Vala Roberts"}, {"prefix": "ka", "field": "email"}, {"field": "manages.reports", "match_phrase": "Beryl Killian Joby Duvessa"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,559 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"conjuncts": [{"field": "name", "match": "Vala Roberts"}, {"prefix": "ka", "field": "email"}, {"field": "manages.reports", "match_phrase": "Beryl Killian Joby Duvessa"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,564 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,564 - root - INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Vala Roberts"}, {"prefix": "ka", "field": "email"}, {"field": "manages.reports", "match_phrase": "Beryl Killian Joby Duvessa"}]} is 0 (took 0.54903ms) 2023-05-20 06:59:59,564 - root - INFO - ES query '{'query': {'bool': {'must': [{'match': {'name': 'Vala Roberts'}}, {'prefix': {'email': 'ka'}}, {'match_phrase': {'manages.reports': 'Beryl Killian Joby Duvessa'}}]}}}' 2023-05-20 06:59:59,574 - root - INFO - ES hits for query: {"query": {"bool": {"must": [{"match": {"name": "Vala Roberts"}}, {"prefix": {"email": "ka"}}, {"match_phrase": {"manages.reports": "Beryl Killian Joby Duvessa"}}]}}} on es_index is 0 (took 7ms) 2023-05-20 06:59:59,574 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,575 - root - INFO - ----------------------------------------------------- Query # 9 ---------------------------------------------------- 2023-05-20 06:59:59,593 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "Temp", "field": "name"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,593 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "Temp", "field": "name"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,598 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,598 - root - INFO - FTS hits for query: {"prefix": "Temp", "field": "name"} is 0 (took 0.218757ms) 2023-05-20 06:59:59,598 - root - INFO - ES query '{'query': {'prefix': {'name': 'Temp'}}}' 2023-05-20 06:59:59,601 - root - INFO - ES hits for query: {"query": {"prefix": {"name": "Temp"}}} on es_index is 0 (took 1ms) 2023-05-20 06:59:59,601 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,602 - root - INFO - ----------------------------------------------------- Query # 10 ---------------------------------------------------- 2023-05-20 06:59:59,617 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must_not": {"disjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Tomás"}]}, "should": {"disjuncts": [{"field": "type", "match": "wiki"}, {"field": "email", "match": "duvessa@mcdiabetes.com"}, {"field": "dept", "match": "Training"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,617 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must_not": {"disjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Tom\xc3\xa1s"}]}, "should": {"disjuncts": [{"field": "type", "match": "wiki"}, {"field": "email", "match": "duvessa@mcdiabetes.com"}, {"field": "dept", "match": "Training"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,625 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,625 - root - INFO - FTS hits for query: {"must_not": {"disjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Tomás"}]}, "should": {"disjuncts": [{"field": "type", "match": "wiki"}, {"field": "email", "match": "duvessa@mcdiabetes.com"}, {"field": "dept", "match": "Training"}]}} is 889 (took 1.022396ms) 2023-05-20 06:59:59,625 - root - INFO - ES query '{'query': {'bool': {'must_not': [{'match': {'dept': 'Engineering'}}, {'match': {'name': 'Tomás'}}], 'should': [{'match': {'type': 'wiki'}}, {'match': {'email': 'duvessa@mcdiabetes.com'}}, {'match': {'dept': 'Training'}}]}}}' 2023-05-20 06:59:59,665 - root - INFO - ES hits for query: {"query": {"bool": {"must_not": [{"match": {"dept": "Engineering"}}, {"match": {"name": "Tomás"}}], "should": [{"match": {"type": "wiki"}}, {"match": {"email": "duvessa@mcdiabetes.com"}}, {"match": {"dept": "Training"}}]}}} on es_index is 889 (took 19ms) 2023-05-20 06:59:59,666 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,666 - root - INFO - ----------------------------------------------------- Query # 11 ---------------------------------------------------- 2023-05-20 06:59:59,683 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+name:\"Vala\" -type:\"emp\" manages.team_size:<9"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,684 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+name:\\"Vala\\" -type:\\"emp\\" manages.team_size:<9"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,689 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,689 - root - INFO - FTS hits for query: {"query": "+name:\"Vala\" -type:\"emp\" manages.team_size:<9"} is 0 (took 1.211078ms) 2023-05-20 06:59:59,689 - root - INFO - ES query '{'query': {'query_string': {'query': '+name:"Vala" -type:"emp" manages.team_size:<9'}}}' 2023-05-20 06:59:59,697 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "+name:\"Vala\" -type:\"emp\" manages.team_size:<9"}}} on es_index is 0 (took 5ms) 2023-05-20 06:59:59,697 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,697 - root - INFO - ----------------------------------------------------- Query # 12 ---------------------------------------------------- 2023-05-20 06:59:59,715 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "w", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,715 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "w", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,720 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,720 - root - INFO - FTS hits for query: {"prefix": "w", "field": "type"} is 0 (took 0.358829ms) 2023-05-20 06:59:59,720 - root - INFO - ES query '{'query': {'prefix': {'type': 'w'}}}' 2023-05-20 06:59:59,723 - root - INFO - ES hits for query: {"query": {"prefix": {"type": "w"}}} on es_index is 0 (took 1ms) 2023-05-20 06:59:59,723 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,724 - root - INFO - ----------------------------------------------------- Query # 13 ---------------------------------------------------- 2023-05-20 06:59:59,740 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1997-03-11T13:52:00", "end": "2023-05-20T06:59:58.168663", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,741 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1997-03-11T13:52:00", "end": "2023-05-20T06:59:58.168663", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,747 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,747 - root - INFO - FTS hits for query: {"field": "join_date", "start": "1997-03-11T13:52:00", "end": "2023-05-20T06:59:58.168663", "inclusive_start": false, "inclusive_end": false} is 301 (took 0.880136ms) 2023-05-20 06:59:59,748 - root - INFO - ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gt': '1997-03-11T13:52:00', 'lt': '2023-05-20T06:59:58.168663'}}}}}}' 2023-05-20 06:59:59,766 - root - INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gt": "1997-03-11T13:52:00", "lt": "2023-05-20T06:59:58.168663"}}}}}} on es_index is 301 (took 11ms) 2023-05-20 06:59:59,767 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,767 - root - INFO - ----------------------------------------------------- Query # 14 ---------------------------------------------------- 2023-05-20 06:59:59,784 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "name", "match": "Gerónimo"}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Deirdre Hall"}, {"field": "languages_known", "match": "German"}]}, "should": {"disjuncts": [{"field": "name", "match": "Amara"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,784 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "name", "match": "Ger\xc3\xb3nimo"}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Deirdre Hall"}, {"field": "languages_known", "match": "German"}]}, "should": {"disjuncts": [{"field": "name", "match": "Amara"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,788 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,789 - root - INFO - FTS hits for query: {"must": {"conjuncts": [{"field": "name", "match": "Gerónimo"}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Deirdre Hall"}, {"field": "languages_known", "match": "German"}]}, "should": {"disjuncts": [{"field": "name", "match": "Amara"}]}} is 10 (took 0.404451ms) 2023-05-20 06:59:59,789 - root - INFO - ES query '{'query': {'bool': {'must': {'match': {'name': 'Gerónimo'}}, 'must_not': [{'match': {'name': 'Deirdre Hall'}}, {'match': {'languages_known': 'German'}}], 'should': {'match': {'name': 'Amara'}}}}}' 2023-05-20 06:59:59,794 - root - INFO - ES hits for query: {"query": {"bool": {"must": {"match": {"name": "Gerónimo"}}, "must_not": [{"match": {"name": "Deirdre Hall"}}, {"match": {"languages_known": "German"}}], "should": {"match": {"name": "Amara"}}}}} on es_index is 10 (took 3ms) 2023-05-20 06:59:59,795 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,795 - root - INFO - ----------------------------------------------------- Query # 15 ---------------------------------------------------- 2023-05-20 06:59:59,817 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,817 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,826 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,827 - root - INFO - FTS hits for query: {"prefix": "emp", "field": "type"} is 1000 (took 0.786646ms) 2023-05-20 06:59:59,827 - root - INFO - ES query '{'query': {'prefix': {'type': 'emp'}}}' 2023-05-20 06:59:59,871 - root - INFO - ES hits for query: {"query": {"prefix": {"type": "emp"}}} on es_index is 1000 (took 22ms) 2023-05-20 06:59:59,871 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,872 - root - INFO - ----------------------------------------------------- Query # 16 ---------------------------------------------------- 2023-05-20 06:59:59,888 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,889 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,900 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,900 - root - INFO - FTS hits for query: {"prefix": "emp", "field": "type"} is 1000 (took 2.966315ms) 2023-05-20 06:59:59,901 - root - INFO - ES query '{'query': {'prefix': {'type': 'emp'}}}' 2023-05-20 06:59:59,938 - root - INFO - ES hits for query: {"query": {"prefix": {"type": "emp"}}} on es_index is 1000 (took 15ms) 2023-05-20 06:59:59,939 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,939 - root - INFO - ----------------------------------------------------- Query # 17 ---------------------------------------------------- 2023-05-20 06:59:59,955 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+manages.team_size:>5 +manages.team_size:<=1186"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 06:59:59,955 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+manages.team_size:>5 +manages.team_size:<=1186"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 06:59:59,962 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 06:59:59,963 - root - INFO - FTS hits for query: {"query": "+manages.team_size:>5 +manages.team_size:<=1186"} is 404 (took 1.034332ms) 2023-05-20 06:59:59,963 - root - INFO - ES query '{'query': {'query_string': {'query': '+manages.team_size:>5 +manages.team_size:<=1186'}}}' 2023-05-20 06:59:59,991 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "+manages.team_size:>5 +manages.team_size:<=1186"}}} on es_index is 404 (took 15ms) 2023-05-20 06:59:59,991 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 06:59:59,992 - root - INFO - ----------------------------------------------------- Query # 18 ---------------------------------------------------- 2023-05-20 07:00:00,006 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Delores"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:00,006 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Delores"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:00,011 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:00,011 - root - INFO - FTS hits for query: {"field": "name", "match": "Delores"} is 19 (took 0.67807ms) 2023-05-20 07:00:00,011 - root - INFO - ES query '{'query': {'match': {'name': 'Delores'}}}' 2023-05-20 07:00:00,016 - root - INFO - ES hits for query: {"query": {"match": {"name": "Delores"}}} on es_index is 19 (took 2ms) 2023-05-20 07:00:00,016 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:00,016 - root - INFO - ----------------------------------------------------- Query # 19 ---------------------------------------------------- 2023-05-20 07:00:00,034 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1975-05-12T23:30:00", "end": "2023-05-20T06:59:58.169507", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:00,034 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1975-05-12T23:30:00", "end": "2023-05-20T06:59:58.169507", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:00,043 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:00,043 - root - INFO - FTS hits for query: {"field": "join_date", "start": "1975-05-12T23:30:00", "end": "2023-05-20T06:59:58.169507", "inclusive_start": true, "inclusive_end": true} is 647 (took 1.030025ms) 2023-05-20 07:00:00,043 - root - INFO - ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gte': '1975-05-12T23:30:00', 'lte': '2023-05-20T06:59:58.169507'}}}}}}' 2023-05-20 07:00:00,071 - root - INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gte": "1975-05-12T23:30:00", "lte": "2023-05-20T06:59:58.169507"}}}}}} on es_index is 647 (took 13ms) 2023-05-20 07:00:00,071 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:00,071 - root - INFO - ----------------------------------------------------- Query # 20 ---------------------------------------------------- 2023-05-20 07:00:00,087 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "-salary:>117000"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:00,087 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "-salary:>117000"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:00,097 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:00,097 - root - INFO - FTS hits for query: {"query": "-salary:>117000"} is 686 (took 2.972907ms) 2023-05-20 07:00:00,097 - root - INFO - ES query '{'query': {'query_string': {'query': '-salary:>117000'}}}' 2023-05-20 07:00:00,125 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "-salary:>117000"}}} on es_index is 686 (took 12ms) 2023-05-20 07:00:00,125 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:00,126 - root - INFO - SUCCESS: 20 out of 20 queries passed 2023-05-20 07:00:00,138 - root - INFO - Node versions in cluster ['6.5.1-6296-enterprise', '6.5.1-6296-enterprise', '6.5.1-6296-enterprise', '6.5.1-6296-enterprise'] 2023-05-20 07:00:00,148 - root - INFO - list of kv nodes in cluster: [ip:172.23.104.98 port:8091 ssh_username:root, ip:172.23.105.123 port:8091 ssh_username:root, ip:172.23.105.191 port:8091 ssh_username:root, ip:172.23.105.44 port:8091 ssh_username:root] 2023-05-20 07:00:00,968 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 07:00:01,133 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 07:00:01,749 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:00:02,184 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:00:02,186 - root - INFO - running command.raw on 172.23.104.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2023-05-20 07:00:02,506 - root - INFO - command executed successfully with root 2023-05-20 07:00:02,508 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.98:8091',default bucket 2023-05-20 07:00:02,510 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 07:00:02,684 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 07:00:03,345 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:00:04,034 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:00:04,035 - root - INFO - running command.raw on 172.23.105.191: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2023-05-20 07:00:04,393 - root - INFO - command executed successfully with root 2023-05-20 07:00:04,395 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.191:8091',default bucket 2023-05-20 07:00:04,398 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 07:00:04,531 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 07:00:04,724 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:00:05,156 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:00:05,157 - root - INFO - running command.raw on 172.23.105.123: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2023-05-20 07:00:05,381 - root - INFO - command executed successfully with root 2023-05-20 07:00:05,382 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.123:8091',default bucket 2023-05-20 07:00:05,384 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 07:00:05,492 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 07:00:05,794 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:00:06,111 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:00:06,112 - root - INFO - running command.raw on 172.23.105.44: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2023-05-20 07:00:06,287 - root - INFO - command executed successfully with root 2023-05-20 07:00:06,288 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.44:8091',default bucket 2023-05-20 07:00:06,312 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.105.191 port:8091 ssh_username:root] 2023-05-20 07:00:06,389 - root - ERROR - http://172.23.107.206:9200/es_index error 404 reason: status: 404, content: b'{"error":"IndexMissingException[[es_index] missing]","status":404}' b'{"error":"IndexMissingException[[es_index] missing]","status":404}' 2023-05-20 07:00:06,450 - root - INFO - Created empty index 'es_index' on Elastic Search node with custom standard analyzer(default) 2023-05-20 07:00:06,579 - root - INFO - Checking if index already exists ... 2023-05-20 07:00:06,579 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=GET params= client_cert=None verify=False 2023-05-20 07:00:06,584 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=DELETE params= client_cert=None verify=False 2023-05-20 07:00:06,594 - root - INFO - Creating fulltext-index index_default on 172.23.105.44 2023-05-20 07:00:06,595 - root - INFO - {"type": "fulltext-index", "name": "index_default", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {}, "sourceParams": {}} 2023-05-20 07:00:06,595 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=PUT params={"type": "fulltext-index", "name": "index_default", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {}, "sourceParams": {}} client_cert=None verify=False 2023-05-20 07:00:06,662 - root - INFO - Index index_default created 2023-05-20 07:00:08,692 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default verb=GET params= client_cert=None verify=False 2023-05-20 07:00:08,792 - root - INFO - Starting operation 'create' on Elastic Search ... 2023-05-20 07:00:08,896 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 07:00:08,998 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 07:00:09,188 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 07:00:09,327 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 07:00:09,701 - root - INFO - 1000 documents bulk loaded into ES 2023-05-20 07:00:09,803 - root - INFO - ES index count for 'es_index': 1000 2023-05-20 07:00:10,643 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/count verb=GET params= client_cert=None verify=False 2023-05-20 07:00:10,665 - root - INFO - http://172.23.104.98:8091/pools/default/buckets/default/stats?zoom=minute 2023-05-20 07:00:10,724 - root - INFO - Docs in bucket = 1000, docs in FTS index 'index_default': 1000, docs in ES index: 1000 2023-05-20 07:00:11,334 - root - INFO - ----------------------------------------------------- Query # 1 ---------------------------------------------------- 2023-05-20 07:00:11,353 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"disjuncts": [{"field": "name", "match": "Solita Aarón"}, {"field": "languages_known", "match": "Urdu"}, {"field": "languages_known", "match": "Quechua"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,353 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"disjuncts": [{"field": "name", "match": "Solita Aar\xc3\xb3n"}, {"field": "languages_known", "match": "Urdu"}, {"field": "languages_known", "match": "Quechua"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,360 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,360 - root - INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Solita Aarón"}, {"field": "languages_known", "match": "Urdu"}, {"field": "languages_known", "match": "Quechua"}]} is 266 (took 0.814585ms) 2023-05-20 07:00:11,360 - root - INFO - ES query '{'query': {'bool': {'should': [{'match': {'name': 'Solita Aarón'}}, {'match': {'languages_known': 'Urdu'}}, {'match': {'languages_known': 'Quechua'}}]}}}' 2023-05-20 07:00:11,381 - root - INFO - ES hits for query: {"query": {"bool": {"should": [{"match": {"name": "Solita Aarón"}}, {"match": {"languages_known": "Urdu"}}, {"match": {"languages_known": "Quechua"}}]}}} on es_index is 266 (took 10ms) 2023-05-20 07:00:11,381 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,382 - root - INFO - ----------------------------------------------------- Query # 2 ---------------------------------------------------- 2023-05-20 07:00:11,398 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\"Malay\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,398 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+languages_known:\\"Malay\\""}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,405 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,405 - root - INFO - FTS hits for query: {"query": "+languages_known:\"Malay\""} is 145 (took 1.227843ms) 2023-05-20 07:00:11,405 - root - INFO - ES query '{'query': {'query_string': {'query': '+languages_known:"Malay"'}}}' 2023-05-20 07:00:11,422 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "+languages_known:\"Malay\""}}} on es_index is 145 (took 10ms) 2023-05-20 07:00:11,423 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,423 - root - INFO - ----------------------------------------------------- Query # 3 ---------------------------------------------------- 2023-05-20 07:00:11,437 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "email", "match": "ciara@mcdiabetes.com"}, {"field": "manages.reports", "match": "Balandria"}, {"field": "type", "match": "wiki"}]}, "should": {"disjuncts": [{"field": "manages.reports", "match": "Joby Desdomna Vala Salina"}, {"field": "languages_known", "match": "Malay"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,438 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "email", "match": "ciara@mcdiabetes.com"}, {"field": "manages.reports", "match": "Balandria"}, {"field": "type", "match": "wiki"}]}, "should": {"disjuncts": [{"field": "manages.reports", "match": "Joby Desdomna Vala Salina"}, {"field": "languages_known", "match": "Malay"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,443 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,443 - root - INFO - FTS hits for query: {"must": {"conjuncts": [{"field": "email", "match": "ciara@mcdiabetes.com"}, {"field": "manages.reports", "match": "Balandria"}, {"field": "type", "match": "wiki"}]}, "should": {"disjuncts": [{"field": "manages.reports", "match": "Joby Desdomna Vala Salina"}, {"field": "languages_known", "match": "Malay"}]}} is 0 (took 0.793253ms) 2023-05-20 07:00:11,444 - root - INFO - ES query '{'query': {'bool': {'must': [{'match': {'email': 'ciara@mcdiabetes.com'}}, {'match': {'manages.reports': 'Balandria'}}, {'match': {'type': 'wiki'}}], 'should': [{'match': {'manages.reports': 'Joby Desdomna Vala Salina'}}, {'match': {'languages_known': 'Malay'}}]}}}' 2023-05-20 07:00:11,448 - root - INFO - ES hits for query: {"query": {"bool": {"must": [{"match": {"email": "ciara@mcdiabetes.com"}}, {"match": {"manages.reports": "Balandria"}}, {"match": {"type": "wiki"}}], "should": [{"match": {"manages.reports": "Joby Desdomna Vala Salina"}}, {"match": {"languages_known": "Malay"}}]}}} on es_index is 0 (took 2ms) 2023-05-20 07:00:11,448 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,448 - root - INFO - ----------------------------------------------------- Query # 4 ---------------------------------------------------- 2023-05-20 07:00:11,466 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "ilith Smith", "fuzziness": 0}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,466 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "ilith Smith", "fuzziness": 0}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,471 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,471 - root - INFO - FTS hits for query: {"field": "name", "match": "ilith Smith", "fuzziness": 0} is 0 (took 0.42905ms) 2023-05-20 07:00:11,472 - root - INFO - ES query '{'query': {'fuzzy': {'name': {'value': 'ilith Smith', 'fuzziness': 0}}}}' 2023-05-20 07:00:11,475 - root - INFO - ES hits for query: {"query": {"fuzzy": {"name": {"value": "ilith Smith", "fuzziness": 0}}}} on es_index is 0 (took 1ms) 2023-05-20 07:00:11,475 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,475 - root - INFO - ----------------------------------------------------- Query # 5 ---------------------------------------------------- 2023-05-20 07:00:11,489 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "wildcard": "e*"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,490 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "type", "wildcard": "e*"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,499 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,500 - root - INFO - FTS hits for query: {"field": "type", "wildcard": "e*"} is 1000 (took 1.686822ms) 2023-05-20 07:00:11,500 - root - INFO - ES query '{'query': {'wildcard': {'type': 'e*'}}}' 2023-05-20 07:00:11,561 - root - INFO - ES hits for query: {"query": {"wildcard": {"type": "e*"}}} on es_index is 1000 (took 24ms) 2023-05-20 07:00:11,561 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,562 - root - INFO - ----------------------------------------------------- Query # 6 ---------------------------------------------------- 2023-05-20 07:00:11,582 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1974-10-08T07:51:00", "end": "2023-05-20T07:00:10.753405", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,583 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1974-10-08T07:51:00", "end": "2023-05-20T07:00:10.753405", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,591 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,591 - root - INFO - FTS hits for query: {"field": "join_date", "start": "1974-10-08T07:51:00", "end": "2023-05-20T07:00:10.753405", "inclusive_start": true, "inclusive_end": true} is 654 (took 1.168247ms) 2023-05-20 07:00:11,591 - root - INFO - ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gte': '1974-10-08T07:51:00', 'lte': '2023-05-20T07:00:10.753405'}}}}}}' 2023-05-20 07:00:11,619 - root - INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gte": "1974-10-08T07:51:00", "lte": "2023-05-20T07:00:10.753405"}}}}}} on es_index is 654 (took 13ms) 2023-05-20 07:00:11,620 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,620 - root - INFO - ----------------------------------------------------- Query # 7 ---------------------------------------------------- 2023-05-20 07:00:11,650 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match_phrase": "Basha"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,651 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "manages.reports", "match_phrase": "Basha"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,656 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,656 - root - INFO - FTS hits for query: {"field": "manages.reports", "match_phrase": "Basha"} is 37 (took 0.500822ms) 2023-05-20 07:00:11,657 - root - INFO - ES query '{'query': {'match_phrase': {'manages.reports': 'Basha'}}}' 2023-05-20 07:00:11,663 - root - INFO - ES hits for query: {"query": {"match_phrase": {"manages.reports": "Basha"}}} on es_index is 37 (took 3ms) 2023-05-20 07:00:11,663 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,663 - root - INFO - ----------------------------------------------------- Query # 8 ---------------------------------------------------- 2023-05-20 07:00:11,687 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"conjuncts": [{"field": "name", "match": "Vala Roberts"}, {"prefix": "ka", "field": "email"}, {"field": "manages.reports", "match_phrase": "Beryl Killian Joby Duvessa"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,688 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"conjuncts": [{"field": "name", "match": "Vala Roberts"}, {"prefix": "ka", "field": "email"}, {"field": "manages.reports", "match_phrase": "Beryl Killian Joby Duvessa"}]}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,694 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,694 - root - INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Vala Roberts"}, {"prefix": "ka", "field": "email"}, {"field": "manages.reports", "match_phrase": "Beryl Killian Joby Duvessa"}]} is 0 (took 0.524444ms) 2023-05-20 07:00:11,694 - root - INFO - ES query '{'query': {'bool': {'must': [{'match': {'name': 'Vala Roberts'}}, {'prefix': {'email': 'ka'}}, {'match_phrase': {'manages.reports': 'Beryl Killian Joby Duvessa'}}]}}}' 2023-05-20 07:00:11,699 - root - INFO - ES hits for query: {"query": {"bool": {"must": [{"match": {"name": "Vala Roberts"}}, {"prefix": {"email": "ka"}}, {"match_phrase": {"manages.reports": "Beryl Killian Joby Duvessa"}}]}}} on es_index is 0 (took 2ms) 2023-05-20 07:00:11,699 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,699 - root - INFO - ----------------------------------------------------- Query # 9 ---------------------------------------------------- 2023-05-20 07:00:11,722 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "Temp", "field": "name"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,722 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "Temp", "field": "name"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,727 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,727 - root - INFO - FTS hits for query: {"prefix": "Temp", "field": "name"} is 0 (took 0.338409ms) 2023-05-20 07:00:11,727 - root - INFO - ES query '{'query': {'prefix': {'name': 'Temp'}}}' 2023-05-20 07:00:11,731 - root - INFO - ES hits for query: {"query": {"prefix": {"name": "Temp"}}} on es_index is 0 (took 1ms) 2023-05-20 07:00:11,731 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,731 - root - INFO - ----------------------------------------------------- Query # 10 ---------------------------------------------------- 2023-05-20 07:00:11,748 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must_not": {"disjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Tomás"}]}, "should": {"disjuncts": [{"field": "type", "match": "wiki"}, {"field": "email", "match": "duvessa@mcdiabetes.com"}, {"field": "dept", "match": "Training"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,748 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must_not": {"disjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Tom\xc3\xa1s"}]}, "should": {"disjuncts": [{"field": "type", "match": "wiki"}, {"field": "email", "match": "duvessa@mcdiabetes.com"}, {"field": "dept", "match": "Training"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,759 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,759 - root - INFO - FTS hits for query: {"must_not": {"disjuncts": [{"field": "dept", "match": "Engineering"}, {"field": "name", "match": "Tomás"}]}, "should": {"disjuncts": [{"field": "type", "match": "wiki"}, {"field": "email", "match": "duvessa@mcdiabetes.com"}, {"field": "dept", "match": "Training"}]}} is 889 (took 1.306104ms) 2023-05-20 07:00:11,759 - root - INFO - ES query '{'query': {'bool': {'must_not': [{'match': {'dept': 'Engineering'}}, {'match': {'name': 'Tomás'}}], 'should': [{'match': {'type': 'wiki'}}, {'match': {'email': 'duvessa@mcdiabetes.com'}}, {'match': {'dept': 'Training'}}]}}}' 2023-05-20 07:00:11,792 - root - INFO - ES hits for query: {"query": {"bool": {"must_not": [{"match": {"dept": "Engineering"}}, {"match": {"name": "Tomás"}}], "should": [{"match": {"type": "wiki"}}, {"match": {"email": "duvessa@mcdiabetes.com"}}, {"match": {"dept": "Training"}}]}}} on es_index is 889 (took 14ms) 2023-05-20 07:00:11,793 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,793 - root - INFO - ----------------------------------------------------- Query # 11 ---------------------------------------------------- 2023-05-20 07:00:11,813 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+name:\"Vala\" -type:\"emp\" manages.team_size:<9"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,814 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+name:\\"Vala\\" -type:\\"emp\\" manages.team_size:<9"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,819 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,820 - root - INFO - FTS hits for query: {"query": "+name:\"Vala\" -type:\"emp\" manages.team_size:<9"} is 0 (took 0.549348ms) 2023-05-20 07:00:11,820 - root - INFO - ES query '{'query': {'query_string': {'query': '+name:"Vala" -type:"emp" manages.team_size:<9'}}}' 2023-05-20 07:00:11,825 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "+name:\"Vala\" -type:\"emp\" manages.team_size:<9"}}} on es_index is 0 (took 3ms) 2023-05-20 07:00:11,825 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,826 - root - INFO - ----------------------------------------------------- Query # 12 ---------------------------------------------------- 2023-05-20 07:00:11,842 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "w", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,843 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "w", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,847 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,847 - root - INFO - FTS hits for query: {"prefix": "w", "field": "type"} is 0 (took 0.255406ms) 2023-05-20 07:00:11,848 - root - INFO - ES query '{'query': {'prefix': {'type': 'w'}}}' 2023-05-20 07:00:11,858 - root - INFO - ES hits for query: {"query": {"prefix": {"type": "w"}}} on es_index is 0 (took 8ms) 2023-05-20 07:00:11,858 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,858 - root - INFO - ----------------------------------------------------- Query # 13 ---------------------------------------------------- 2023-05-20 07:00:11,878 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1997-03-11T13:52:00", "end": "2023-05-20T07:00:10.754730", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,878 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1997-03-11T13:52:00", "end": "2023-05-20T07:00:10.754730", "inclusive_start": false, "inclusive_end": false}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,885 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,885 - root - INFO - FTS hits for query: {"field": "join_date", "start": "1997-03-11T13:52:00", "end": "2023-05-20T07:00:10.754730", "inclusive_start": false, "inclusive_end": false} is 301 (took 0.705444ms) 2023-05-20 07:00:11,885 - root - INFO - ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gt': '1997-03-11T13:52:00', 'lt': '2023-05-20T07:00:10.754730'}}}}}}' 2023-05-20 07:00:11,902 - root - INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gt": "1997-03-11T13:52:00", "lt": "2023-05-20T07:00:10.754730"}}}}}} on es_index is 301 (took 7ms) 2023-05-20 07:00:11,902 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,902 - root - INFO - ----------------------------------------------------- Query # 14 ---------------------------------------------------- 2023-05-20 07:00:11,922 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "name", "match": "Gerónimo"}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Deirdre Hall"}, {"field": "languages_known", "match": "German"}]}, "should": {"disjuncts": [{"field": "name", "match": "Amara"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,923 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"must": {"conjuncts": [{"field": "name", "match": "Ger\xc3\xb3nimo"}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Deirdre Hall"}, {"field": "languages_known", "match": "German"}]}, "should": {"disjuncts": [{"field": "name", "match": "Amara"}]}}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,928 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,928 - root - INFO - FTS hits for query: {"must": {"conjuncts": [{"field": "name", "match": "Gerónimo"}]}, "must_not": {"disjuncts": [{"field": "name", "match": "Deirdre Hall"}, {"field": "languages_known", "match": "German"}]}, "should": {"disjuncts": [{"field": "name", "match": "Amara"}]}} is 10 (took 0.363358ms) 2023-05-20 07:00:11,928 - root - INFO - ES query '{'query': {'bool': {'must': {'match': {'name': 'Gerónimo'}}, 'must_not': [{'match': {'name': 'Deirdre Hall'}}, {'match': {'languages_known': 'German'}}], 'should': {'match': {'name': 'Amara'}}}}}' 2023-05-20 07:00:11,934 - root - INFO - ES hits for query: {"query": {"bool": {"must": {"match": {"name": "Gerónimo"}}, "must_not": [{"match": {"name": "Deirdre Hall"}}, {"match": {"languages_known": "German"}}], "should": {"match": {"name": "Amara"}}}}} on es_index is 10 (took 3ms) 2023-05-20 07:00:11,934 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:11,934 - root - INFO - ----------------------------------------------------- Query # 15 ---------------------------------------------------- 2023-05-20 07:00:11,951 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:11,951 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:11,960 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:11,960 - root - INFO - FTS hits for query: {"prefix": "emp", "field": "type"} is 1000 (took 0.793582ms) 2023-05-20 07:00:11,961 - root - INFO - ES query '{'query': {'prefix': {'type': 'emp'}}}' 2023-05-20 07:00:12,195 - root - INFO - ES hits for query: {"query": {"prefix": {"type": "emp"}}} on es_index is 1000 (took 11ms) 2023-05-20 07:00:12,196 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:12,196 - root - INFO - ----------------------------------------------------- Query # 16 ---------------------------------------------------- 2023-05-20 07:00:12,213 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:12,214 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"prefix": "emp", "field": "type"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:12,293 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:12,293 - root - INFO - FTS hits for query: {"prefix": "emp", "field": "type"} is 1000 (took 0.867152ms) 2023-05-20 07:00:12,293 - root - INFO - ES query '{'query': {'prefix': {'type': 'emp'}}}' 2023-05-20 07:00:12,326 - root - INFO - ES hits for query: {"query": {"prefix": {"type": "emp"}}} on es_index is 1000 (took 12ms) 2023-05-20 07:00:12,327 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:12,327 - root - INFO - ----------------------------------------------------- Query # 17 ---------------------------------------------------- 2023-05-20 07:00:12,343 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+manages.team_size:>5 +manages.team_size:<=1186"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:12,343 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "+manages.team_size:>5 +manages.team_size:<=1186"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:12,350 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:12,351 - root - INFO - FTS hits for query: {"query": "+manages.team_size:>5 +manages.team_size:<=1186"} is 404 (took 1.087415ms) 2023-05-20 07:00:12,351 - root - INFO - ES query '{'query': {'query_string': {'query': '+manages.team_size:>5 +manages.team_size:<=1186'}}}' 2023-05-20 07:00:12,371 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "+manages.team_size:>5 +manages.team_size:<=1186"}}} on es_index is 404 (took 8ms) 2023-05-20 07:00:12,371 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:12,371 - root - INFO - ----------------------------------------------------- Query # 18 ---------------------------------------------------- 2023-05-20 07:00:12,387 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Delores"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:12,388 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "name", "match": "Delores"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:12,392 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:12,393 - root - INFO - FTS hits for query: {"field": "name", "match": "Delores"} is 19 (took 0.344975ms) 2023-05-20 07:00:12,393 - root - INFO - ES query '{'query': {'match': {'name': 'Delores'}}}' 2023-05-20 07:00:12,397 - root - INFO - ES hits for query: {"query": {"match": {"name": "Delores"}}} on es_index is 19 (took 2ms) 2023-05-20 07:00:12,398 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:12,398 - root - INFO - ----------------------------------------------------- Query # 19 ---------------------------------------------------- 2023-05-20 07:00:12,414 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1975-05-12T23:30:00", "end": "2023-05-20T07:00:10.755575", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:12,414 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"field": "join_date", "start": "1975-05-12T23:30:00", "end": "2023-05-20T07:00:10.755575", "inclusive_start": true, "inclusive_end": true}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:12,422 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:12,422 - root - INFO - FTS hits for query: {"field": "join_date", "start": "1975-05-12T23:30:00", "end": "2023-05-20T07:00:10.755575", "inclusive_start": true, "inclusive_end": true} is 647 (took 1.481236ms) 2023-05-20 07:00:12,422 - root - INFO - ES query '{'query': {'filtered': {'filter': {'range': {'join_date': {'gte': '1975-05-12T23:30:00', 'lte': '2023-05-20T07:00:10.755575'}}}}}}' 2023-05-20 07:00:12,449 - root - INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gte": "1975-05-12T23:30:00", "lte": "2023-05-20T07:00:10.755575"}}}}}} on es_index is 647 (took 9ms) 2023-05-20 07:00:12,450 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:12,450 - root - INFO - ----------------------------------------------------- Query # 20 ---------------------------------------------------- 2023-05-20 07:00:12,467 - root - INFO - Running query {"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "-salary:>117000"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}} on node as 172.23.105.44 : Administrator: 2023-05-20 07:00:12,467 - root - INFO - Making a rest request api=http://172.23.105.44:8094/api/index/index_default/query verb=POST params=b'{"indexName": "index_default", "size": 10000000, "from": 0, "explain": false, "query": {"query": "-salary:>117000"}, "fields": [], "ctl": {"consistency": {"level": "", "vectors": {}}, "timeout": 60000}}' client_cert=None verify=False 2023-05-20 07:00:12,474 - root - INFO - Status: {'total': 6, 'failed': 0, 'successful': 6} 2023-05-20 07:00:12,474 - root - INFO - FTS hits for query: {"query": "-salary:>117000"} is 686 (took 1.10031ms) 2023-05-20 07:00:12,474 - root - INFO - ES query '{'query': {'query_string': {'query': '-salary:>117000'}}}' 2023-05-20 07:00:12,499 - root - INFO - ES hits for query: {"query": {"query_string": {"query": "-salary:>117000"}}} on es_index is 686 (took 9ms) 2023-05-20 07:00:12,500 - root - INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified 2023-05-20 07:00:12,501 - root - INFO - SUCCESS: 20 out of 20 queries passed 2023-05-20 07:00:12,537 - root - INFO - 'None' bucket's settings will be changed with parameters: parallelDBAndViewCompaction=false&databaseFragmentationThreshold%5Bpercentage%5D=20&viewFragmentationThreshold%5Bpercentage%5D=20 2023-05-20 07:00:12,543 - root - INFO - Indexer Compaction Settings: {'indexCompactionMode': 'full', 'indexCircularCompaction[interval][fromHour]': 0, 'indexCircularCompaction[interval][fromMinute]': 0, 'indexCircularCompaction[interval][toHour]': 0, 'indexCircularCompaction[interval][toMinute]': 0, 'indexCircularCompaction[interval][abortOutside]': 'false', 'parallelDBAndViewCompaction': 'false', 'indexFragmentationThreshold[percentage]': 20} 2023-05-20 07:00:12,573 - root - INFO - sleep for 10 secs. ... 2023-05-20 07:00:22,583 - root - INFO - Create index with replica and query 2023-05-20 07:00:22,600 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.105.191 port:8091 ssh_username:root] 2023-05-20 07:00:22,611 - root - INFO - list of n1ql nodes in cluster: [ip:172.23.105.191 port:8091 ssh_username:root] 2023-05-20 07:00:22,612 - root - INFO - create_index 2023-05-20 07:00:22,612 - root - INFO - Check if index existed in default on server 172.23.105.191 2023-05-20 07:00:22,619 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2023-05-20 07:00:22,627 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2023-05-20 07:00:23,025 - root - INFO - TOTAL ELAPSED TIME: 394.410433ms 2023-05-20 07:00:23,025 - root - ERROR - Fail to get index list. List output: {'requestID': '6cd7a456-39a7-4d17-ac7e-d88d6da096a7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '394.410433ms', 'executionTime': '394.20494ms', 'resultCount': 0, 'resultSize': 0}} 2023-05-20 07:00:23,025 - root - INFO - Create primary index 2023-05-20 07:00:23,033 - root - INFO - RUN QUERY CREATE PRIMARY INDEX ON default USING GSI 2023-05-20 07:00:23,043 - root - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default++USING+GSI 2023-05-20 07:00:28,172 - root - INFO - TOTAL ELAPSED TIME: 5.125871874s 2023-05-20 07:00:28,173 - root - INFO - Check if index is online 2023-05-20 07:00:28,184 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2023-05-20 07:00:28,194 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2023-05-20 07:00:28,227 - root - INFO - TOTAL ELAPSED TIME: 29.736875ms 2023-05-20 07:00:28,227 - root - INFO - done create_index 2023-05-20 07:00:28,234 - root - INFO - RUN QUERY CREATE INDEX random_index_986491 ON default(age) USING GSI WITH {'num_replica': 2}; 2023-05-20 07:00:28,241 - root - INFO - query params : statement=CREATE+INDEX+random_index_986491+ON+default%28age%29+USING+GSI+WITH+%7B%27num_replica%27%3A+2%7D%3B 2023-05-20 07:00:32,022 - root - INFO - TOTAL ELAPSED TIME: 3.777793562s 2023-05-20 07:00:32,022 - root - INFO - sleep for 30 secs. ... 2023-05-20 07:01:02,062 - root - INFO - Making a rest request api=http://172.23.104.98:8091/indexStatus verb=GET params= client_cert=None verify=False 2023-05-20 07:01:02,068 - root - INFO - {'default': {'#primary': {'status': 'Ready', 'progress': '100', 'definition': 'CREATE PRIMARY INDEX `#primary` ON `default`', 'hosts': '172.23.105.44:8091', 'id': 5807157434358931667}, 'random_index_986491': {'status': 'Ready', 'progress': '100', 'definition': 'CREATE INDEX `random_index_986491` ON `default`(`age`) WITH { "nodes":[ "172.23.105.123:8091","172.23.105.191:8091","172.23.105.44:8091" ], "num_replica":2 }', 'hosts': '172.23.105.123:8091', 'id': 12776130655355069287}, 'random_index_986491 (replica 1)': {'status': 'Ready', 'progress': '100', 'definition': 'CREATE INDEX `random_index_986491` ON `default`(`age`) WITH { "nodes":[ "172.23.105.123:8091","172.23.105.191:8091","172.23.105.44:8091" ], "num_replica":2 }', 'hosts': '172.23.105.44:8091', 'id': 12776130655355069287}, 'random_index_986491 (replica 2)': {'status': 'Ready', 'progress': '100', 'definition': 'CREATE INDEX `random_index_986491` ON `default`(`age`) WITH { "nodes":[ "172.23.105.123:8091","172.23.105.191:8091","172.23.105.44:8091" ], "num_replica":2 }', 'hosts': '172.23.105.191:8091', 'id': 12776130655355069287}}} 2023-05-20 07:01:02,068 - root - INFO - Hostnames : 172.23.105.123:8091 , 172.23.105.44:8091 2023-05-20 07:01:02,068 - root - INFO - Index IDs : 12776130655355069287, 12776130655355069287 2023-05-20 07:01:02,068 - root - INFO - Hostnames : 172.23.105.123:8091 , 172.23.105.191:8091 2023-05-20 07:01:02,069 - root - INFO - Index IDs : 12776130655355069287, 12776130655355069287 2023-05-20 07:01:02,069 - root - INFO - waiting for memcached bucket : default in 172.23.104.98 to accept set ops 2023-05-20 07:01:02,242 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 07:01:02,429 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 07:01:02,549 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 07:01:02,673 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 07:01:02,874 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 07:01:02,990 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 07:01:03,164 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 07:01:03,274 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 07:01:04,048 - root - INFO - creating direct client 172.23.104.98:11210 default 2023-05-20 07:01:04,162 - root - INFO - creating direct client 172.23.105.123:11210 default 2023-05-20 07:01:04,338 - root - INFO - creating direct client 172.23.105.191:11210 default 2023-05-20 07:01:04,445 - root - INFO - creating direct client 172.23.105.44:11210 default 2023-05-20 07:01:04,507 - root - INFO - sleep for 15 secs. Pre-setup of old version is done. Wait for upgrade to 7.2.0-5325 version ... 2023-05-20 07:01:29,536 - root - INFO - rebalance progress took 10.01 seconds 2023-05-20 07:01:29,537 - root - INFO - sleep for 10 seconds after rebalance... rebalance status: True 2023-05-20 07:01:39,602 - root - INFO - fail_over node ns_1@172.23.105.44 successful 2023-05-20 07:01:39,602 - root - INFO - sleep for 60 secs. ... 2023-05-20 07:02:39,627 - root - INFO - Going to set recoveryType=full for node :: ns_1@172.23.105.44 2023-05-20 07:02:39,638 - root - INFO - recoveryType for node ns_1@172.23.105.44 set successful 2023-05-20 07:02:39,640 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 07:02:39,751 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 07:02:40,046 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:02:40,359 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:02:40,360 - root - INFO - finding build 7.2.0-5325 for machine ip:172.23.105.44 port:8091 ssh_username:root 2023-05-20 07:02:40,361 - root - INFO - running command.raw on 172.23.105.44: /sbin/iptables -F 2023-05-20 07:02:40,418 - root - INFO - command executed successfully with root 2023-05-20 07:02:40,418 - root - INFO - running command.raw on 172.23.105.44: /sbin/iptables -t nat -F 2023-05-20 07:02:40,473 - root - INFO - command executed successfully with root 2023-05-20 07:02:40,474 - root - INFO - SSH Connecting to 172.23.105.44 with username:root, attempt#1 of 5 2023-05-20 07:02:40,577 - root - INFO - SSH Connected to 172.23.105.44 as root 2023-05-20 07:02:40,876 - root - INFO - running command.raw on 172.23.105.44: cd /tmp;wget -nc -q -O couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/5325/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 07:02:42,030 - root - INFO - command executed successfully with root 2023-05-20 07:02:42,030 - root - INFO - total 451M 2023-05-20 07:02:42,031 - root - INFO - -rw-r--r--. 1 root root 451M May 17 20:56 couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:02:42,031 - root - INFO - drwx------. 3 root root 16 Mar 13 09:01 systemd-private-356447aa087e462d9c6fbb42a35324e1-ntpd.service-43UyhJ 2023-05-20 07:02:42,055 - root - INFO - deliverable_type : rpm 2023-05-20 07:02:42,055 - root - INFO - /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm or /tmp/ 2023-05-20 07:02:42,055 - root - INFO - running command.raw on 172.23.105.44: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:03:49,470 - root - INFO - command executed successfully with root 2023-05-20 07:03:49,471 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2023-05-20 07:03:49,471 - root - INFO - running command.raw on 172.23.105.44: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 07:03:49,533 - root - INFO - command executed successfully with root 2023-05-20 07:03:49,533 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:03:49,619 - root - INFO - process beam.smp is running on 172.23.105.44: with pid 347 2023-05-20 07:03:49,730 - root - INFO - Running systemd command on this server 2023-05-20 07:03:49,730 - root - INFO - running command.raw on 172.23.105.44: systemctl stop couchbase-server.service 2023-05-20 07:03:50,661 - root - INFO - command executed successfully with root 2023-05-20 07:03:50,661 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:03:50,733 - root - INFO - Starting couchbase server 2023-05-20 07:03:50,839 - root - INFO - Running systemd command on this server 2023-05-20 07:03:50,839 - root - INFO - running command.raw on 172.23.105.44: systemctl start couchbase-server.service 2023-05-20 07:03:50,857 - root - INFO - command executed successfully with root 2023-05-20 07:03:50,857 - root - INFO - 172.23.105.44:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 07:03:55,863 - root - INFO - running command.raw on 172.23.105.44: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 07:03:55,895 - root - INFO - command executed successfully with root 2023-05-20 07:03:55,895 - root - INFO - Couchbase server status: [' Process: 415 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=1/FAILURE)'] 2023-05-20 07:03:55,895 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:03:55,970 - root - INFO - process beam.smp is running on 172.23.105.44: with pid 460 2023-05-20 07:03:55,970 - root - INFO - upgrade 172.23.105.44 to version 7.2.0-5325 is completed 2023-05-20 07:03:56,033 - root - INFO - 172.23.105.44 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 07:03:56,106 - root - INFO - Running systemd start command on this server 2023-05-20 07:03:56,106 - root - INFO - running command.raw on 172.23.105.44: systemctl start couchbase-server.service 2023-05-20 07:03:56,128 - root - INFO - command executed successfully with root 2023-05-20 07:03:56,129 - root - INFO - sleep for 30 secs. wait for servers up ... 2023-05-20 07:04:26,169 - root - INFO - waiting for ns_server @ 172.23.105.44:8091 2023-05-20 07:04:26,170 - root - INFO - -->is_ns_server_running? 2023-05-20 07:04:26,173 - root - INFO - ns_server @ 172.23.105.44:8091 is running 2023-05-20 07:04:26,173 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 07:04:26,174 - root - INFO - settings/web params on 172.23.105.44:8091:port=8091&username=Administrator&password=password 2023-05-20 07:04:26,246 - root - INFO - --> status:True 2023-05-20 07:04:26,247 - root - INFO - sleep for 15 secs. ... 2023-05-20 07:04:41,256 - root - INFO - sleep for 10 secs. ... 2023-05-20 07:04:51,631 - root - INFO - Node 172.23.105.44 not part of cluster inactiveAdded 2023-05-20 07:04:51,654 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.123,ns_1@172.23.105.191,ns_1@172.23.105.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 07:04:51,669 - root - INFO - rebalance operation started 2023-05-20 07:05:01,688 - root - INFO - rebalance percentage : 23.00 % 2023-05-20 07:05:01,688 - root - INFO - Rebalance - status: running, progress: 23.00% 2023-05-20 07:05:21,734 - root - INFO - Rebalance - status: none, progress: 100.00% 2023-05-20 07:05:21,745 - root - INFO - rebalancing was completed with progress: 100% in 30.075809478759766 sec 2023-05-20 07:05:31,771 - root - INFO - rebalance progress took 10.03 seconds 2023-05-20 07:05:31,771 - root - INFO - sleep for 10 seconds after rebalance... rebalance status: True 2023-05-20 07:05:41,818 - root - INFO - fail_over node ns_1@172.23.105.123 successful 2023-05-20 07:05:41,818 - root - INFO - sleep for 60 secs. ... 2023-05-20 07:06:41,825 - root - INFO - Going to set recoveryType=full for node :: ns_1@172.23.105.123 2023-05-20 07:06:41,832 - root - INFO - recoveryType for node ns_1@172.23.105.123 set successful 2023-05-20 07:06:41,834 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 07:06:42,001 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 07:06:42,203 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:06:42,659 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:06:42,660 - root - INFO - finding build 7.2.0-5325 for machine ip:172.23.105.123 port:8091 ssh_username:root 2023-05-20 07:06:42,660 - root - INFO - running command.raw on 172.23.105.123: /sbin/iptables -F 2023-05-20 07:06:42,739 - root - INFO - command executed successfully with root 2023-05-20 07:06:42,740 - root - INFO - running command.raw on 172.23.105.123: /sbin/iptables -t nat -F 2023-05-20 07:06:42,816 - root - INFO - command executed successfully with root 2023-05-20 07:06:42,817 - root - INFO - SSH Connecting to 172.23.105.123 with username:root, attempt#1 of 5 2023-05-20 07:06:42,946 - root - INFO - SSH Connected to 172.23.105.123 as root 2023-05-20 07:06:43,153 - root - INFO - running command.raw on 172.23.105.123: cd /tmp;wget -nc -q -O couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/5325/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 07:06:47,460 - root - INFO - command executed successfully with root 2023-05-20 07:06:47,460 - root - INFO - total 451M 2023-05-20 07:06:47,461 - root - INFO - -rw-r--r-- 1 root root 451M May 17 20:56 couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:06:47,461 - root - INFO - drwxrwxrwx 2 root root 6 May 20 05:30 sg_logs 2023-05-20 07:06:47,461 - root - INFO - drwx------ 3 root root 16 Apr 19 07:49 systemd-private-3ebe73983dad4ffe83894d045f6584e6-ntpd.service-hcBhNh 2023-05-20 07:06:47,551 - root - INFO - deliverable_type : rpm 2023-05-20 07:06:47,551 - root - INFO - /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm or /tmp/ 2023-05-20 07:06:47,552 - root - INFO - running command.raw on 172.23.105.123: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:07:52,250 - root - INFO - command executed successfully with root 2023-05-20 07:07:52,252 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2023-05-20 07:07:52,252 - root - INFO - running command.raw on 172.23.105.123: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 07:07:52,314 - root - INFO - command executed successfully with root 2023-05-20 07:07:52,315 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:07:52,387 - root - INFO - process beam.smp is running on 172.23.105.123: with pid 128072 2023-05-20 07:07:52,558 - root - INFO - Running systemd command on this server 2023-05-20 07:07:52,558 - root - INFO - running command.raw on 172.23.105.123: systemctl stop couchbase-server.service 2023-05-20 07:07:53,452 - root - INFO - command executed successfully with root 2023-05-20 07:07:53,453 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:07:53,513 - root - INFO - Starting couchbase server 2023-05-20 07:07:53,696 - root - INFO - Running systemd command on this server 2023-05-20 07:07:53,696 - root - INFO - running command.raw on 172.23.105.123: systemctl start couchbase-server.service 2023-05-20 07:07:53,739 - root - INFO - command executed successfully with root 2023-05-20 07:07:53,740 - root - INFO - 172.23.105.123:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 07:07:58,745 - root - INFO - running command.raw on 172.23.105.123: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 07:07:58,796 - root - INFO - command executed successfully with root 2023-05-20 07:07:58,796 - root - INFO - Couchbase server status: [' Process: 128133 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=1/FAILURE)'] 2023-05-20 07:07:58,797 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:07:58,857 - root - INFO - process beam.smp is running on 172.23.105.123: with pid 128166 2023-05-20 07:07:58,857 - root - INFO - upgrade 172.23.105.123 to version 7.2.0-5325 is completed 2023-05-20 07:07:58,942 - root - INFO - 172.23.105.123 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 07:07:59,069 - root - INFO - Running systemd start command on this server 2023-05-20 07:07:59,070 - root - INFO - running command.raw on 172.23.105.123: systemctl start couchbase-server.service 2023-05-20 07:07:59,105 - root - INFO - command executed successfully with root 2023-05-20 07:07:59,105 - root - INFO - sleep for 30 secs. wait for servers up ... 2023-05-20 07:08:29,139 - root - INFO - waiting for ns_server @ 172.23.105.123:8091 2023-05-20 07:08:29,140 - root - INFO - -->is_ns_server_running? 2023-05-20 07:08:29,146 - root - INFO - ns_server @ 172.23.105.123:8091 is running 2023-05-20 07:08:29,146 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 07:08:29,146 - root - INFO - settings/web params on 172.23.105.123:8091:port=8091&username=Administrator&password=password 2023-05-20 07:08:29,239 - root - INFO - --> status:True 2023-05-20 07:08:29,239 - root - INFO - sleep for 15 secs. ... 2023-05-20 07:08:44,254 - root - INFO - sleep for 10 secs. ... 2023-05-20 07:08:54,996 - root - INFO - Node 172.23.105.123 not part of cluster inactiveAdded 2023-05-20 07:08:55,039 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.123,ns_1@172.23.105.191,ns_1@172.23.105.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 07:08:55,046 - root - INFO - rebalance operation started 2023-05-20 07:09:05,080 - root - INFO - rebalance percentage : 25.00 % 2023-05-20 07:09:05,080 - root - INFO - Rebalance - status: running, progress: 25.00% 2023-05-20 07:09:25,140 - root - INFO - Rebalance - status: none, progress: 100.00% 2023-05-20 07:09:25,154 - root - INFO - rebalancing was completed with progress: 100% in 30.10774278640747 sec 2023-05-20 07:09:35,173 - root - INFO - rebalance progress took 10.02 seconds 2023-05-20 07:09:35,173 - root - INFO - sleep for 10 seconds after rebalance... rebalance status: True 2023-05-20 07:09:45,224 - root - INFO - fail_over node ns_1@172.23.105.191 successful 2023-05-20 07:09:45,224 - root - INFO - sleep for 60 secs. ... 2023-05-20 07:10:45,256 - root - INFO - Going to set recoveryType=full for node :: ns_1@172.23.105.191 2023-05-20 07:10:45,274 - root - INFO - recoveryType for node ns_1@172.23.105.191 set successful 2023-05-20 07:10:45,276 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 07:10:45,468 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 07:10:46,175 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:10:46,855 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:10:46,856 - root - INFO - finding build 7.2.0-5325 for machine ip:172.23.105.191 port:8091 ssh_username:root 2023-05-20 07:10:46,856 - root - INFO - running command.raw on 172.23.105.191: /sbin/iptables -F 2023-05-20 07:10:46,930 - root - INFO - command executed successfully with root 2023-05-20 07:10:46,930 - root - INFO - running command.raw on 172.23.105.191: /sbin/iptables -t nat -F 2023-05-20 07:10:47,003 - root - INFO - command executed successfully with root 2023-05-20 07:10:47,004 - root - INFO - SSH Connecting to 172.23.105.191 with username:root, attempt#1 of 5 2023-05-20 07:10:47,168 - root - INFO - SSH Connected to 172.23.105.191 as root 2023-05-20 07:10:47,777 - root - INFO - running command.raw on 172.23.105.191: cd /tmp;wget -nc -q -O couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/5325/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 07:10:52,057 - root - INFO - command executed successfully with root 2023-05-20 07:10:52,058 - root - INFO - total 451M 2023-05-20 07:10:52,058 - root - INFO - -rw-r--r--. 1 root root 451M May 17 20:56 couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:10:52,058 - root - INFO - drwxr-xr-x. 3 couchbase couchbase 20 May 19 19:04 data 2023-05-20 07:10:52,058 - root - INFO - drwxr-xr-x. 4 couchbase couchbase 35 May 19 18:57 index 2023-05-20 07:10:52,058 - root - INFO - drwx------. 3 root root 16 Mar 28 08:17 systemd-private-4490784fc3cf419ea9ec0ce02ea4eb49-ntpd.service-QDU0x1 2023-05-20 07:10:52,138 - root - INFO - deliverable_type : rpm 2023-05-20 07:10:52,138 - root - INFO - /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm or /tmp/ 2023-05-20 07:10:52,138 - root - INFO - running command.raw on 172.23.105.191: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:13:07,307 - root - INFO - command executed successfully with root 2023-05-20 07:13:07,309 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2023-05-20 07:13:07,309 - root - INFO - running command.raw on 172.23.105.191: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 07:13:07,440 - root - INFO - command executed successfully with root 2023-05-20 07:13:07,440 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:13:07,560 - root - INFO - process beam.smp is running on 172.23.105.191: with pid 23795 2023-05-20 07:13:07,816 - root - INFO - Running systemd command on this server 2023-05-20 07:13:07,816 - root - INFO - running command.raw on 172.23.105.191: systemctl stop couchbase-server.service 2023-05-20 07:13:08,601 - root - INFO - command executed successfully with root 2023-05-20 07:13:08,602 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:13:08,699 - root - INFO - Starting couchbase server 2023-05-20 07:13:08,981 - root - INFO - Running systemd command on this server 2023-05-20 07:13:08,981 - root - INFO - running command.raw on 172.23.105.191: systemctl start couchbase-server.service 2023-05-20 07:13:09,060 - root - INFO - command executed successfully with root 2023-05-20 07:13:09,061 - root - INFO - 172.23.105.191:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 07:13:14,066 - root - INFO - running command.raw on 172.23.105.191: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 07:13:14,162 - root - INFO - command executed successfully with root 2023-05-20 07:13:14,162 - root - INFO - Couchbase server status: [' Process: 23900 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=1/FAILURE)'] 2023-05-20 07:13:14,163 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:13:14,268 - root - INFO - process beam.smp is running on 172.23.105.191: with pid 23967 2023-05-20 07:13:14,269 - root - INFO - upgrade 172.23.105.191 to version 7.2.0-5325 is completed 2023-05-20 07:13:14,385 - root - INFO - 172.23.105.191 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 07:13:14,595 - root - INFO - Running systemd start command on this server 2023-05-20 07:13:14,596 - root - INFO - running command.raw on 172.23.105.191: systemctl start couchbase-server.service 2023-05-20 07:13:14,675 - root - INFO - command executed successfully with root 2023-05-20 07:13:14,675 - root - INFO - sleep for 30 secs. wait for servers up ... 2023-05-20 07:13:44,723 - root - INFO - waiting for ns_server @ 172.23.105.191:8091 2023-05-20 07:13:44,724 - root - INFO - -->is_ns_server_running? 2023-05-20 07:13:44,730 - root - INFO - ns_server @ 172.23.105.191:8091 is running 2023-05-20 07:13:44,730 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 07:13:44,730 - root - INFO - settings/web params on 172.23.105.191:8091:port=8091&username=Administrator&password=password 2023-05-20 07:13:44,884 - root - INFO - --> status:True 2023-05-20 07:13:44,884 - root - INFO - sleep for 15 secs. ... 2023-05-20 07:13:59,900 - root - INFO - sleep for 10 secs. ... 2023-05-20 07:14:10,483 - root - INFO - Node 172.23.105.191 not part of cluster inactiveAdded 2023-05-20 07:14:10,523 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.123,ns_1@172.23.105.191,ns_1@172.23.105.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 07:14:10,531 - root - INFO - rebalance operation started 2023-05-20 07:14:20,549 - root - INFO - rebalance percentage : 9.00 % 2023-05-20 07:14:20,549 - root - INFO - Rebalance - status: running, progress: 9.00% 2023-05-20 07:14:40,588 - root - INFO - Rebalance - status: none, progress: 100.00% 2023-05-20 07:14:40,602 - root - INFO - rebalancing was completed with progress: 100% in 30.07079315185547 sec 2023-05-20 07:14:50,621 - root - INFO - rebalance progress took 10.02 seconds 2023-05-20 07:14:50,621 - root - INFO - sleep for 10 seconds after rebalance... rebalance status: True 2023-05-20 07:15:00,673 - root - INFO - fail_over node ns_1@172.23.104.98 successful 2023-05-20 07:15:00,674 - root - INFO - sleep for 60 secs. ... 2023-05-20 07:16:00,720 - root - INFO - Going to set recoveryType=full for node :: ns_1@172.23.104.98 2023-05-20 07:16:00,732 - root - INFO - recoveryType for node ns_1@172.23.104.98 set successful 2023-05-20 07:16:00,734 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 07:16:00,894 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 07:16:01,468 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2023-05-20 07:16:01,904 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2023-05-20 07:16:01,905 - root - INFO - finding build 7.2.0-5325 for machine ip:172.23.104.98 port:8091 ssh_username:root 2023-05-20 07:16:01,905 - root - INFO - running command.raw on 172.23.104.98: /sbin/iptables -F 2023-05-20 07:16:01,982 - root - INFO - command executed successfully with root 2023-05-20 07:16:01,982 - root - INFO - running command.raw on 172.23.104.98: /sbin/iptables -t nat -F 2023-05-20 07:16:02,061 - root - INFO - command executed successfully with root 2023-05-20 07:16:02,061 - root - INFO - SSH Connecting to 172.23.104.98 with username:root, attempt#1 of 5 2023-05-20 07:16:02,239 - root - INFO - SSH Connected to 172.23.104.98 as root 2023-05-20 07:16:02,926 - root - INFO - running command.raw on 172.23.104.98: cd /tmp;wget -nc -q -O couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/5325/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm;cd /tmp;ls -lh 2023-05-20 07:16:07,102 - root - INFO - command executed successfully with root 2023-05-20 07:16:07,102 - root - INFO - total 451M 2023-05-20 07:16:07,102 - root - INFO - -rw-r--r--. 1 root root 451M May 17 20:56 couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:16:07,102 - root - INFO - drwx------. 3 root root 17 May 19 11:00 systemd-private-a21e5daf5f9f4891bbed337ba41a1ca3-ntpd.service-8BDkU9 2023-05-20 07:16:07,190 - root - INFO - deliverable_type : rpm 2023-05-20 07:16:07,190 - root - INFO - /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm or /tmp/ 2023-05-20 07:16:07,190 - root - INFO - running command.raw on 172.23.104.98: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.2.0-5325-centos7.x86_64.rpm 2023-05-20 07:18:12,920 - root - INFO - command executed successfully with root 2023-05-20 07:18:12,921 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2023-05-20 07:18:12,921 - root - INFO - running command.raw on 172.23.104.98: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2023-05-20 07:18:13,011 - root - INFO - command executed successfully with root 2023-05-20 07:18:13,011 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:18:13,148 - root - INFO - process beam.smp is running on 172.23.104.98: with pid 4306 2023-05-20 07:18:13,319 - root - INFO - Running systemd command on this server 2023-05-20 07:18:13,319 - root - INFO - running command.raw on 172.23.104.98: systemctl stop couchbase-server.service 2023-05-20 07:18:14,038 - root - INFO - command executed successfully with root 2023-05-20 07:18:14,039 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:18:14,100 - root - INFO - Starting couchbase server 2023-05-20 07:18:14,276 - root - INFO - Running systemd command on this server 2023-05-20 07:18:14,276 - root - INFO - running command.raw on 172.23.104.98: systemctl start couchbase-server.service 2023-05-20 07:18:14,322 - root - INFO - command executed successfully with root 2023-05-20 07:18:14,322 - root - INFO - 172.23.104.98:sleep for 5 secs. waiting for couchbase server to come up ... 2023-05-20 07:18:19,328 - root - INFO - running command.raw on 172.23.104.98: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2023-05-20 07:18:19,387 - root - INFO - command executed successfully with root 2023-05-20 07:18:19,387 - root - INFO - Couchbase server status: [' Process: 4406 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=1/FAILURE)'] 2023-05-20 07:18:19,388 - root - INFO - Checking for process beam.smp on linux 2023-05-20 07:18:19,478 - root - INFO - process beam.smp is running on 172.23.104.98: with pid 4448 2023-05-20 07:18:19,478 - root - INFO - upgrade 172.23.104.98 to version 7.2.0-5325 is completed 2023-05-20 07:18:19,551 - root - INFO - 172.23.104.98 **** The linux version file /opt/couchbase/ VERSION.txt exists 2023-05-20 07:18:19,668 - root - INFO - Running systemd start command on this server 2023-05-20 07:18:19,668 - root - INFO - running command.raw on 172.23.104.98: systemctl start couchbase-server.service 2023-05-20 07:18:19,705 - root - INFO - command executed successfully with root 2023-05-20 07:18:19,705 - root - INFO - sleep for 30 secs. wait for servers up ... 2023-05-20 07:18:49,750 - root - INFO - waiting for ns_server @ 172.23.104.98:8091 2023-05-20 07:18:49,750 - root - INFO - -->is_ns_server_running? 2023-05-20 07:18:49,758 - root - INFO - ns_server @ 172.23.104.98:8091 is running 2023-05-20 07:18:49,758 - root - INFO - --> in init_cluster...Administrator,password,8091 2023-05-20 07:18:49,758 - root - INFO - settings/web params on 172.23.104.98:8091:port=8091&username=Administrator&password=password 2023-05-20 07:18:49,906 - root - INFO - --> status:True 2023-05-20 07:18:49,906 - root - INFO - sleep for 15 secs. ... 2023-05-20 07:19:04,921 - root - INFO - sleep for 10 secs. ... 2023-05-20 07:19:15,921 - root - INFO - Node 172.23.104.98 not part of cluster inactiveAdded 2023-05-20 07:19:15,968 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.98,ns_1@172.23.105.123,ns_1@172.23.105.191,ns_1@172.23.105.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2023-05-20 07:19:16,040 - root - INFO - rebalance operation started 2023-05-20 07:19:26,056 - root - INFO - rebalance percentage : 28.00 % 2023-05-20 07:19:26,056 - root - INFO - Rebalance - status: running, progress: 28.00% 2023-05-20 07:19:46,100 - root - INFO - rebalance percentage : 54.00 % 2023-05-20 07:19:46,100 - root - INFO - Rebalance - status: running, progress: 54.00% 2023-05-20 07:20:06,138 - root - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2023-05-20 07:20:06,166 - root - INFO - Latest logs from UI on 172.23.104.98: 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1684592404405, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {agent_died,<20089.29558.1>,\n {linked_process_died,<20089.29559.1>,\n {\'ns_1@172.23.105.123\',\n {no_connection,"index-service_api"}}}}}.\nRebalance Operation Id = f08c91e2cdf57b6b931ebe8fb4ba7312', 'serverTime': '2023-05-20T07:20:04.405Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 0, 'module': 'ns_vbucket_mover', 'tstamp': 1684592358607, 'shortText': 'message', 'text': 'Bucket "default" rebalance does not seem to be swap rebalance', 'serverTime': '2023-05-20T07:19:18.607Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.104.98', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1684592356642, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.104.98\' in 0 seconds.', 'serverTime': '2023-05-20T07:19:16.642Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 0, 'module': 'ns_rebalancer', 'tstamp': 1684592356438, 'shortText': 'message', 'text': 'Started rebalancing bucket default', 'serverTime': '2023-05-20T07:19:16.438Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.104.98', 'type': 'info', 'code': 0, 'module': 'ns_storage_conf', 'tstamp': 1684592356247, 'shortText': 'message', 'text': 'Deleting old data files of bucket "default"', 'serverTime': '2023-05-20T07:19:16.247Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1684592355975, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.104.98','ns_1@172.23.105.123',\n 'ns_1@172.23.105.191','ns_1@172.23.105.44'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f08c91e2cdf57b6b931ebe8fb4ba7312", 'serverTime': '2023-05-20T07:19:15.975Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.105.191', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1684592315102, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.105.191\' in 0 seconds.', 'serverTime': '2023-05-20T07:18:35.102Z'} 2023-05-20 07:20:06,167 - root - ERROR - {'node': 'ns_1@172.23.105.191', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1684592314981, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"collections_enabled">>]', 'serverTime': '2023-05-20T07:18:34.981Z'} 2023-05-20 07:20:06,168 - root - ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1684592314946, 'shortText': 'message', 'text': 'Enabled auto-failover with timeout 120 and max count 1', 'serverTime': '2023-05-20T07:18:34.946Z'} 2023-05-20 07:20:06,168 - root - ERROR - {'node': 'ns_1@172.23.105.44', 'type': 'info', 'code': 0, 'module': 'ns_janitor_server', 'tstamp': 1684592314945, 'shortText': 'message', 'text': 'The following topology-aware services will be upgraded during the next rebalance: [n1ql]', 'serverTime': '2023-05-20T07:18:34.945Z'} [, , , , , ] Sat May 20 07:20:06 2023 [, , , , , , , , , ] 2023-05-20 07:20:06,184 - root - INFO - ============== EventingBaseTest tearDown has started ============== 2023-05-20 07:20:06,206 - root - INFO - cannot find service node eventing in cluster 2023-05-20 07:20:06,245 - root - INFO - ------- Cluster statistics ------- 2023-05-20 07:20:06,246 - root - INFO - 172.23.104.98:8091 => {'services': ['kv'], 'cpu_utilization': 1.860924027862374, 'mem_free': 3145842688, 'mem_total': 3963314176, 'swap_mem_used': 0, 'swap_mem_total': 0} 2023-05-20 07:20:06,246 - root - INFO - 172.23.105.123:8091 => {'services': ['index', 'kv'], 'cpu_utilization': 0, 'mem_free': 0, 'mem_total': 0, 'swap_mem_used': 0, 'swap_mem_total': 0} 2023-05-20 07:20:06,246 - root - INFO - 172.23.105.191:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.13084927035983, 'mem_free': 2958901248, 'mem_total': 3963314176, 'swap_mem_used': 36630528, 'swap_mem_total': 3758092288} 2023-05-20 07:20:06,246 - root - INFO - 172.23.105.44:8091 => {'services': ['fts', 'index', 'kv'], 'cpu_utilization': 3.245444057785194, 'mem_free': 2786693120, 'mem_total': 4201582592, 'swap_mem_used': 570425344, 'swap_mem_total': 3758092288} 2023-05-20 07:20:06,246 - root - INFO - --- End of cluster statistics --- 2023-05-20 07:20:06,246 - root - INFO - ============== basetestcase cleanup was started for test #1 test_offline_upgrade_with_add_new_services ============== 2023-05-20 07:20:06,357 - root - INFO - deleting existing buckets ['default'] on 172.23.104.98 Build was aborted Aborted by Ashwin G Archiving artifacts *** TestRunner *** Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion No emails were triggered. Finished: ABORTED