/Users/lfc/venv/bin/python "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 49836 --file /Users/lfc/testrunner/testrunner.py -i b/resources/tmp.ini -p gsi_type=plasma,initial_version=6.6.5-10067,upgrade_to=7.1.0-2074,get-cbcollect-info=True,java_sdk_client=True -c conf/gsi/collection-upgrade-gsi.conf Connected to pydev debugger (build 212.4746.96) /Users/lfc/testrunner/lib/builds/build_query.py:818: SyntaxWarning: "is not" with a literal. Did you mean "!="? if toy is not "": /Users/lfc/testrunner/lib/builds/build_query.py:938: SyntaxWarning: "is not" with a literal. Did you mean "!="? elif toy is not "": /Users/lfc/testrunner/./lib/remote/remote_util.py:4225: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': /Users/lfc/testrunner/lib/remote/remote_util.py:4225: SyntaxWarning: "is not" with a literal. Did you mean "!="? if command_options is not '': INFO:root:__main__ *** TestRunner *** INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: conf/gsi/collection-upgrade-gsi.conf INFO:__main__:Test prefix: gsi.upgrade_gsi.UpgradeSecondaryIndex INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: {'cluster_name': 'tmp', 'conf_file': 'conf/gsi/collection-upgrade-gsi.conf', 'get-cbcollect-info': 'True', 'gsi_type': 'plasma', 'ini': 'b/resources/tmp.ini', 'initial_version': '6.6.5-10067', 'java_sdk_client': 'True', 'num_nodes': 6, 'spec': 'collection-upgrade-gsi', 'upgrade_to': '7.1.0-2074'} INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: gsi.upgrade_gsi.UpgradeSecondaryIndex.test_offline_upgrade,nodes_init=5,services_init=kv-index-index-kv-n1ql,standard_buckets=1,doc_ops=True,doc-per-day=10,dataset=default,scan_consistency=request_plus,groups=simple,init_nodes=False,build_index_after_create=False,num_index_replica=1 INFO:__main__:Logs folder: /Users/lfc/testrunner/logs/testrunner-22-Feb-03_12-09-09/test_1 Logs will be stored at /Users/lfc/testrunner/logs/testrunner-22-Feb-03_12-09-09/test_1 ./testrunner -i b/resources/tmp.ini -p gsi_type=plasma,initial_version=6.6.5-10067,upgrade_to=7.1.0-2074,get-cbcollect-info=True,java_sdk_client=True -t gsi.upgrade_gsi.UpgradeSecondaryIndex.test_offline_upgrade,nodes_init=5,services_init=kv-index-index-kv-n1ql,standard_buckets=1,doc_ops=True,doc-per-day=10,dataset=default,scan_consistency=request_plus,groups=simple,init_nodes=False,build_index_after_create=False,num_index_replica=1 Test Input params: {'nodes_init': '5', 'services_init': 'kv-index-index-kv-n1ql', 'standard_buckets': '1', 'doc_ops': 'True', 'doc-per-day': '10', 'dataset': 'default', 'scan_consistency': 'request_plus', 'groups': 'simple', 'init_nodes': 'False', 'build_index_after_create': 'False', 'num_index_replica': '1', 'ini': 'b/resources/tmp.ini', 'cluster_name': 'tmp', 'spec': 'collection-upgrade-gsi', 'conf_file': 'conf/gsi/collection-upgrade-gsi.conf', 'gsi_type': 'plasma', 'initial_version': '6.6.5-10067', 'upgrade_to': '7.1.0-2074', 'get-cbcollect-info': 'True', 'java_sdk_client': 'True', 'num_nodes': 6, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/Users/lfc/testrunner/logs/testrunner-22-Feb-03_12-09-09/test_1'} /Users/lfc/testrunner/pytests/failover/AutoFailoverBaseTest.py:617: SyntaxWarning: "is" with a literal. Did you mean "=="? if self.failover_action is "restart_server": /Users/lfc/testrunner/pytests/basetestcase.py:1490: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.stats_monitor is not '': /Users/lfc/testrunner/pytests/basetestcase.py:1492: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.warmup_stats_monitor is not '': /Users/lfc/testrunner/lib/tasks/task.py:302: SyntaxWarning: "is" with a literal. Did you mean "=="? if bucket_params['bucket_priority'] is None or bucket_params['bucket_priority'].lower() is 'low': /Users/lfc/testrunner/lib/membase/helper/cluster_helper.py:382: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/lfc/testrunner/./lib/membase/helper/cluster_helper.py:382: SyntaxWarning: "is" with a literal. Did you mean "=="? if index_path is '': /Users/lfc/testrunner/lib/couchbase_helper/tuq_helper.py:219: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/lfc/testrunner/lib/couchbase_helper/tuq_helper.py:229: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/lfc/testrunner/scripts/install.py:301: SyntaxWarning: "is not" with a literal. Did you mean "!="? if toy is not "": /Users/lfc/testrunner/./pytests/basetestcase.py:1490: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.stats_monitor is not '': /Users/lfc/testrunner/./pytests/basetestcase.py:1492: SyntaxWarning: "is not" with a literal. Did you mean "!="? if self.warmup_stats_monitor is not '': /Users/lfc/testrunner/./lib/couchbase_helper/tuq_helper.py:219: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/lfc/testrunner/./lib/couchbase_helper/tuq_helper.py:229: SyntaxWarning: "is" with a literal. Did you mean "=="? if status is "any": /Users/lfc/testrunner/pytests/fts/random_query_generator/rand_query_gen.py:1311: SyntaxWarning: "is" with a literal. Did you mean "=="? while fieldname is "conjunction_disjunction": fieldname = self.get_random_value(self.query_types) /Users/lfc/testrunner/./pytests/fts/random_query_generator/rand_query_gen.py:1311: SyntaxWarning: "is" with a literal. Did you mean "=="? while fieldname is "conjunction_disjunction": fieldname = self.get_random_value(self.query_types) test_offline_upgrade (gsi.upgrade_gsi.UpgradeSecondaryIndex) Offline Upgrade. ... 2022-02-03 12:09:10,221 - root - INFO - ============== EventingBaseTest setup has started ============== 2022-02-03 12:09:10,224 - root - INFO - Starting Test: test_offline_upgrade Offline Upgrade. 1) Perform Operations 2) Stop cb service on all nodes. 3) Upgrade all nodes. 4) Start cb service on all nodes. 5) Perform Operations 2022-02-03 12:09:10,230 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:09:10,389 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:09:10,488 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:10,741 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:10,757 - root - INFO - -->is_ns_server_running? 2022-02-03 12:09:10,772 - root - INFO - Node version in cluster 6.6.5-10067-enterprise 2022-02-03 12:09:10,773 - root - INFO - ============== basetestcase setup was started for test #1 test_offline_upgrade============== 2022-02-03 12:09:10,781 - root - ERROR - GET http://10.112.205.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:10,803 - root - ERROR - GET http://10.112.205.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:10,805 - root - INFO - cannot find service node index in cluster 2022-02-03 12:09:10,809 - root - INFO - Building docker image with java sdk client 2022-02-03 12:09:16,368 - root - INFO - initializing cluster 2022-02-03 12:09:16,832 - root - INFO - server: ip:10.112.205.101 port:8091 ssh_username:root, nodes/self 2022-02-03 12:09:16,837 - root - INFO - {'uptime': '26', 'memoryTotal': 1930829824, 'memoryFree': 1586503680, 'mcdMemoryReserved': 1473, 'mcdMemoryAllocated': 1473, 'status': 'healthy', 'hostname': '10.112.205.101:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10067-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 256, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.101', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1841, 'curr_items': 0} 2022-02-03 12:09:16,842 - root - ERROR - GET http://10.112.205.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:16,844 - root - INFO - pools/default params : memoryQuota=986 2022-02-03 12:09:16,850 - root - INFO - --> init_node_services(Administrator,password,10.112.205.101,8091,['kv']) 2022-02-03 12:09:16,851 - root - INFO - /node/controller/setupServices params on 10.112.205.101: 8091:hostname=10.112.205.101&user=Administrator&password=password&services=kv 2022-02-03 12:09:16,866 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:09:16,867 - root - INFO - settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2022-02-03 12:09:16,915 - root - INFO - --> status:True 2022-02-03 12:09:16,919 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:09:17,025 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:09:17,243 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:17,558 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:17,560 - root - INFO - running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:09:17,777 - root - INFO - command executed successfully with root 2022-02-03 12:09:17,778 - root - INFO - ['ok'] 2022-02-03 12:09:17,786 - root - INFO - /diag/eval status on 10.112.205.101:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:17,790 - root - INFO - /diag/eval status on 10.112.205.101:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:17,795 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-03 12:09:17,818 - root - INFO - server: ip:10.112.205.102 port:8091 ssh_username:root, nodes/self 2022-02-03 12:09:17,823 - root - INFO - {'uptime': '24', 'memoryTotal': 2033594368, 'memoryFree': 1686781952, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.102:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10067-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.102', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-03 12:09:17,826 - root - ERROR - GET http://10.112.205.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:17,828 - root - INFO - pools/default params : memoryQuota=1039 2022-02-03 12:09:17,839 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:09:17,840 - root - INFO - settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2022-02-03 12:09:17,887 - root - INFO - --> status:True 2022-02-03 12:09:17,890 - root - INFO - SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-03 12:09:18,110 - root - INFO - SSH Connected to 10.112.205.102 as root 2022-02-03 12:09:18,352 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:18,613 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:18,615 - root - INFO - running command.raw on 10.112.205.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:09:18,820 - root - INFO - command executed successfully with root 2022-02-03 12:09:18,821 - root - INFO - ['ok'] 2022-02-03 12:09:18,831 - root - INFO - /diag/eval status on 10.112.205.102:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:18,836 - root - INFO - /diag/eval status on 10.112.205.102:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:18,842 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-03 12:09:18,871 - root - INFO - server: ip:10.112.205.103 port:8091 ssh_username:root, nodes/self 2022-02-03 12:09:18,877 - root - INFO - {'uptime': '25', 'memoryTotal': 2033594368, 'memoryFree': 1691054080, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.103:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10067-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.103', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-03 12:09:18,883 - root - ERROR - GET http://10.112.205.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:18,885 - root - INFO - pools/default params : memoryQuota=1039 2022-02-03 12:09:18,898 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:09:18,899 - root - INFO - settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2022-02-03 12:09:18,961 - root - INFO - --> status:True 2022-02-03 12:09:18,965 - root - INFO - SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-03 12:09:19,084 - root - INFO - SSH Connected to 10.112.205.103 as root 2022-02-03 12:09:19,363 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:19,669 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:19,671 - root - INFO - running command.raw on 10.112.205.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:09:19,883 - root - INFO - command executed successfully with root 2022-02-03 12:09:19,884 - root - INFO - ['ok'] 2022-02-03 12:09:19,895 - root - INFO - /diag/eval status on 10.112.205.103:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:19,899 - root - INFO - /diag/eval status on 10.112.205.103:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:19,905 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-03 12:09:19,929 - root - INFO - server: ip:10.112.205.104 port:8091 ssh_username:root, nodes/self 2022-02-03 12:09:19,937 - root - INFO - {'uptime': '30', 'memoryTotal': 2033594368, 'memoryFree': 1690460160, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.104:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10067-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.104', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-03 12:09:19,942 - root - ERROR - GET http://10.112.205.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:19,944 - root - INFO - pools/default params : memoryQuota=1039 2022-02-03 12:09:19,954 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:09:19,955 - root - INFO - settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2022-02-03 12:09:20,012 - root - INFO - --> status:True 2022-02-03 12:09:20,016 - root - INFO - SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-03 12:09:20,117 - root - INFO - SSH Connected to 10.112.205.104 as root 2022-02-03 12:09:20,398 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:20,691 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:20,693 - root - INFO - running command.raw on 10.112.205.104: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:09:20,906 - root - INFO - command executed successfully with root 2022-02-03 12:09:20,907 - root - INFO - ['ok'] 2022-02-03 12:09:20,919 - root - INFO - /diag/eval status on 10.112.205.104:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:20,925 - root - INFO - /diag/eval status on 10.112.205.104:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:20,931 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-03 12:09:20,960 - root - INFO - server: ip:10.112.205.105 port:8091 ssh_username:root, nodes/self 2022-02-03 12:09:20,966 - root - INFO - {'uptime': '30', 'memoryTotal': 2033594368, 'memoryFree': 1692119040, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.105:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10067-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.105', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-03 12:09:20,972 - root - ERROR - GET http://10.112.205.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:20,974 - root - INFO - pools/default params : memoryQuota=1039 2022-02-03 12:09:20,982 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:09:20,984 - root - INFO - settings/web params on 10.112.205.105:8091:port=8091&username=Administrator&password=password 2022-02-03 12:09:21,045 - root - INFO - --> status:True 2022-02-03 12:09:21,048 - root - INFO - SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-03 12:09:21,167 - root - INFO - SSH Connected to 10.112.205.105 as root 2022-02-03 12:09:21,445 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:21,738 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:21,739 - root - INFO - running command.raw on 10.112.205.105: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:09:21,955 - root - INFO - command executed successfully with root 2022-02-03 12:09:21,957 - root - INFO - ['ok'] 2022-02-03 12:09:21,965 - root - INFO - /diag/eval status on 10.112.205.105:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:21,968 - root - INFO - /diag/eval status on 10.112.205.105:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:21,973 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-03 12:09:21,998 - root - INFO - server: ip:10.112.205.106 port:8091 ssh_username:root, nodes/self 2022-02-03 12:09:22,003 - root - INFO - {'uptime': '30', 'memoryTotal': 2033594368, 'memoryFree': 1694429184, 'mcdMemoryReserved': 1551, 'mcdMemoryAllocated': 1551, 'status': 'healthy', 'hostname': '10.112.205.106:8091', 'clusterCompatibility': 393222, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '6.6.5-10067-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 271, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '10.112.205.106', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 1939, 'curr_items': 0} 2022-02-03 12:09:22,008 - root - ERROR - GET http://10.112.205.106:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-02-03 12:09:22,011 - root - INFO - pools/default params : memoryQuota=1039 2022-02-03 12:09:22,019 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:09:22,020 - root - INFO - settings/web params on 10.112.205.106:8091:port=8091&username=Administrator&password=password 2022-02-03 12:09:22,073 - root - INFO - --> status:True 2022-02-03 12:09:22,076 - root - INFO - SSH Connecting to 10.112.205.106 with username:root, attempt#1 of 5 2022-02-03 12:09:22,192 - root - INFO - SSH Connected to 10.112.205.106 as root 2022-02-03 12:09:22,466 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:09:22,765 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:09:22,767 - root - INFO - running command.raw on 10.112.205.106: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:09:22,983 - root - INFO - command executed successfully with root 2022-02-03 12:09:22,984 - root - INFO - ['ok'] 2022-02-03 12:09:22,994 - root - INFO - /diag/eval status on 10.112.205.106:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:22,997 - root - INFO - /diag/eval status on 10.112.205.106:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). 2022-02-03 12:09:23,002 - root - INFO - settings/indexes params : storageMode=plasma 2022-02-03 12:09:23,022 - root - INFO - **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2022-02-03 12:09:23,042 - root - ERROR - DELETE http://10.112.205.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-03 12:09:23,045 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2022-02-03 12:09:23,100 - root - INFO - sleep for 5 secs. ... 2022-02-03 12:09:28,106 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2022-02-03 12:09:28,116 - root - INFO - done initializing cluster 2022-02-03 12:09:29,046 - root - INFO - adding node 10.112.205.102:8091 to cluster 2022-02-03 12:09:29,052 - root - INFO - adding remote node @10.112.205.102:18091 to this cluster @10.112.205.101:8091 2022-02-03 12:09:39,060 - root - INFO - rebalance progress took 10.01 seconds 2022-02-03 12:09:39,061 - root - INFO - sleep for 10 seconds after rebalance... 2022-02-03 12:09:50,767 - root - INFO - adding node 10.112.205.103:8091 to cluster 2022-02-03 12:09:50,774 - root - INFO - adding remote node @10.112.205.103:18091 to this cluster @10.112.205.101:8091 2022-02-03 12:10:00,783 - root - INFO - rebalance progress took 10.01 seconds 2022-02-03 12:10:00,784 - root - INFO - sleep for 10 seconds after rebalance... 2022-02-03 12:10:12,519 - root - INFO - adding node 10.112.205.104:8091 to cluster 2022-02-03 12:10:12,526 - root - INFO - adding remote node @10.112.205.104:18091 to this cluster @10.112.205.101:8091 2022-02-03 12:10:22,537 - root - INFO - rebalance progress took 10.01 seconds 2022-02-03 12:10:22,537 - root - INFO - sleep for 10 seconds after rebalance... 2022-02-03 12:10:34,222 - root - INFO - adding node 10.112.205.105:8091 to cluster 2022-02-03 12:10:34,229 - root - INFO - adding remote node @10.112.205.105:18091 to this cluster @10.112.205.101:8091 2022-02-03 12:10:44,241 - root - INFO - rebalance progress took 10.01 seconds 2022-02-03 12:10:44,242 - root - INFO - sleep for 10 seconds after rebalance... 2022-02-03 12:10:55,853 - root - INFO - Node 10.112.205.102 not part of cluster inactiveAdded 2022-02-03 12:10:55,854 - root - INFO - Node 10.112.205.103 not part of cluster inactiveAdded 2022-02-03 12:10:55,854 - root - INFO - Node 10.112.205.104 not part of cluster inactiveAdded 2022-02-03 12:10:55,855 - root - INFO - Node 10.112.205.105 not part of cluster inactiveAdded 2022-02-03 12:10:55,866 - root - INFO - rebalance params : {'knownNodes': 'ns_1@10.112.205.101,ns_1@10.112.205.102,ns_1@10.112.205.103,ns_1@10.112.205.104,ns_1@10.112.205.105', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-02-03 12:10:55,876 - root - INFO - rebalance operation started 2022-02-03 12:11:05,885 - root - INFO - Rebalance - status: none, progress: 100.00% 2022-02-03 12:11:05,891 - root - INFO - rebalancing was completed with progress: 100% in 10.014293193817139 sec 2022-02-03 12:11:05,894 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:11:05,950 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:11:06,031 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:06,323 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:06,325 - root - INFO - running command.raw on 10.112.205.101: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-03 12:11:06,530 - root - INFO - command executed successfully with root 2022-02-03 12:11:06,532 - root - INFO - ['ok'] 2022-02-03 12:11:06,533 - root - INFO - Enabled diag/eval for non-local hosts from 10.112.205.101 2022-02-03 12:11:06,910 - root - INFO - http://10.112.205.101:8091/pools/default/buckets with param: name=standard_bucket0&ramQuotaMB=986&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-02-03 12:11:06,921 - root - INFO - 0.01 seconds to create bucket standard_bucket0 2022-02-03 12:11:06,923 - root - INFO - waiting for memcached bucket : standard_bucket0 in 10.112.205.101 to accept set ops 2022-02-03 12:11:07,644 - root - INFO - creating direct client 10.112.205.101:11210 standard_bucket0 2022-02-03 12:11:07,712 - root - INFO - creating direct client 10.112.205.104:11210 standard_bucket0 2022-02-03 12:11:07,841 - root - INFO - creating direct client 10.112.205.101:11210 standard_bucket0 2022-02-03 12:11:07,910 - root - INFO - creating direct client 10.112.205.104:11210 standard_bucket0 2022-02-03 12:11:09,152 - root - INFO - bucket 'standard_bucket0' was created with per node RAM quota: 986 2022-02-03 12:11:09,153 - root - INFO - ============== basetestcase setup was finished for test #1 test_offline_upgrade ============== 2022-02-03 12:11:09,162 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:11:09,215 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:11:09,322 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:09,561 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:09,564 - root - INFO - SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-03 12:11:09,637 - root - INFO - SSH Connected to 10.112.205.102 as root 2022-02-03 12:11:09,709 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:09,984 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:09,986 - root - INFO - SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-03 12:11:10,063 - root - INFO - SSH Connected to 10.112.205.103 as root 2022-02-03 12:11:10,141 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:10,391 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:10,394 - root - INFO - SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-03 12:11:10,524 - root - INFO - SSH Connected to 10.112.205.104 as root 2022-02-03 12:11:10,610 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:10,859 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:10,862 - root - INFO - SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-03 12:11:10,946 - root - INFO - SSH Connected to 10.112.205.105 as root 2022-02-03 12:11:11,030 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:11,279 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:11,283 - root - INFO - SSH Connecting to 10.112.205.106 with username:root, attempt#1 of 5 2022-02-03 12:11:11,340 - root - INFO - SSH Connected to 10.112.205.106 as root 2022-02-03 12:11:11,427 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:11,684 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:15,944 - root - INFO - ------- Cluster statistics ------- 2022-02-03 12:11:15,945 - root - INFO - 10.112.205.101:8091 => {'services': ['kv'], 'cpu_utilization': 0, 'mem_free': 1472651264, 'mem_total': 1930829824, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-02-03 12:11:15,946 - root - INFO - 10.112.205.102:8091 => {'services': ['index'], 'cpu_utilization': 1.01010101010101, 'mem_free': 1603604480, 'mem_total': 2033594368, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-02-03 12:11:15,947 - root - INFO - 10.112.205.103:8091 => {'services': ['index'], 'cpu_utilization': 0, 'mem_free': 1585573888, 'mem_total': 2033594368, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-02-03 12:11:15,947 - root - INFO - 10.112.205.104:8091 => {'services': ['kv'], 'cpu_utilization': 1.030927835051546, 'mem_free': 1590628352, 'mem_total': 2033594368, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-02-03 12:11:15,948 - root - INFO - 10.112.205.105:8091 => {'services': ['n1ql'], 'cpu_utilization': 37.9746835443038, 'mem_free': 1621532672, 'mem_total': 2033594368, 'swap_mem_used': 0, 'swap_mem_total': 1107292160} 2022-02-03 12:11:15,949 - root - INFO - --- End of cluster statistics --- 2022-02-03 12:11:15,949 - root - INFO - Time to execute basesetup : 125.72366690635681 2022-02-03 12:11:15,956 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:11:16,010 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:11:16,092 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:16,348 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:18,497 - root - INFO - list of n1ql nodes in cluster: [ip:10.112.205.105 port:8091 ssh_username:root] 2022-02-03 12:11:18,504 - root - INFO - cannot find service node eventing in cluster 2022-02-03 12:11:18,509 - root - INFO - list of n1ql nodes in cluster: [ip:10.112.205.105 port:8091 ssh_username:root] 2022-02-03 12:11:18,513 - root - INFO - ============== EventingBaseTest setup has completed ============== 2022-02-03 12:11:18,516 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:11:18,611 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:11:18,696 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:18,946 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:21,099 - root - INFO - list of n1ql nodes in cluster: [ip:10.112.205.105 port:8091 ssh_username:root] 2022-02-03 12:11:21,102 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:11:21,155 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:11:21,237 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:21,488 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:21,529 - root - INFO - Initial status of 10.112.205.101 cluster is healthy 2022-02-03 12:11:21,530 - root - INFO - current status of 10.112.205.101 is healthy 2022-02-03 12:11:21,542 - root - INFO - Initial status of 10.112.205.102 cluster is healthy 2022-02-03 12:11:21,543 - root - INFO - current status of 10.112.205.102 is healthy 2022-02-03 12:11:21,557 - root - INFO - Initial status of 10.112.205.103 cluster is healthy 2022-02-03 12:11:21,558 - root - INFO - current status of 10.112.205.103 is healthy 2022-02-03 12:11:21,571 - root - INFO - Initial status of 10.112.205.104 cluster is healthy 2022-02-03 12:11:21,572 - root - INFO - current status of 10.112.205.104 is healthy 2022-02-03 12:11:21,584 - root - INFO - Initial status of 10.112.205.105 cluster is healthy 2022-02-03 12:11:21,585 - root - INFO - current status of 10.112.205.105 is healthy 2022-02-03 12:11:21,597 - root - INFO - Initial status of 10.112.205.106 cluster is healthy 2022-02-03 12:11:21,598 - root - INFO - current status of 10.112.205.106 is healthy 2022-02-03 12:11:21,606 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:11:21,624 - root - INFO - {'indexer.settings.storage_mode': 'plasma'} set 2022-02-03 12:11:21,626 - root - INFO - Allowing the indexer to complete restart after setting the internal settings 2022-02-03 12:11:21,628 - root - INFO - sleep for 5 secs. ... 2022-02-03 12:11:26,641 - root - INFO - {'indexer.api.enableTestServer': True} set 2022-02-03 12:11:26,645 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:11:26,732 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:11:26,812 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:11:27,104 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:11:30,648 - root - INFO - create 20160.0 to standard_bucket0 documents... 2022-02-03 12:11:30,693 - root - INFO - creating direct client 10.112.205.101:11210 standard_bucket0 2022-02-03 12:11:30,845 - root - INFO - creating direct client 10.112.205.104:11210 standard_bucket0 2022-02-03 12:11:47,643 - root - INFO - LOAD IS FINISHED 2022-02-03 12:11:47,647 - root - INFO - {'remaining': {'start': 0, 'end': 10}} 2022-02-03 12:11:49,803 - root - INFO - list of n1ql nodes in cluster: [ip:10.112.205.105 port:8091 ssh_username:root] 2022-02-03 12:11:49,804 - root - INFO - ip:10.112.205.105 port:8091 ssh_username:root 2022-02-03 12:11:49,804 - root - INFO - sleep for 30 secs. ... 2022-02-03 12:12:19,810 - root - INFO - Check if index existed in standard_bucket0 on server 10.112.205.105 2022-02-03 12:12:19,817 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2022-02-03 12:12:19,822 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2022-02-03 12:12:19,963 - root - INFO - TOTAL ELAPSED TIME: 137.581748ms 2022-02-03 12:12:19,964 - root - ERROR - Fail to get index list. List output: {'requestID': '8fca44a5-fe70-4dfc-835a-a591ed85ac28', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '137.581748ms', 'executionTime': '137.528935ms', 'resultCount': 0, 'resultSize': 0}} 2022-02-03 12:12:19,965 - root - INFO - Create primary index 2022-02-03 12:12:19,973 - root - INFO - RUN QUERY CREATE PRIMARY INDEX ON standard_bucket0 with {"num_replica": 1 } 2022-02-03 12:12:19,980 - root - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket0++with+%7B%22num_replica%22%3A+1+%7D 2022-02-03 12:12:22,530 - root - INFO - TOTAL ELAPSED TIME: 2.546981454s 2022-02-03 12:12:22,531 - root - INFO - Check if index is online 2022-02-03 12:12:22,536 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2022-02-03 12:12:22,543 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2022-02-03 12:12:22,564 - root - INFO - TOTAL ELAPSED TIME: 17.113742ms 2022-02-03 12:12:22,577 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:12:22,637 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:12:22,723 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:22,975 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:22,997 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:12:23,003 - root - INFO - Making a rest request api=http://10.112.205.102:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False 2022-02-03 12:12:23,039 - root - INFO - {'queryport.client.waitForScheduledIndex': False} set 2022-02-03 12:12:23,041 - root - INFO - Making a rest request api=http://10.112.205.102:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False 2022-02-03 12:12:23,065 - root - INFO - {'indexer.allowScheduleCreateRebal': True} set 2022-02-03 12:12:23,997 - root - INFO - RUN QUERY CREATE INDEX `index_name_0` ON standard_bucket0(job_title) USING GSI WITH {'defer_build': True, 'num_replica': 1} 2022-02-03 12:12:24,002 - root - INFO - query params : statement=CREATE+INDEX+%60index_name_0%60+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27defer_build%27%3A+True%2C+%27num_replica%27%3A+1%7D 2022-02-03 12:12:24,136 - root - INFO - TOTAL ELAPSED TIME: 129.957406ms 2022-02-03 12:12:24,138 - root - INFO - BUILD INDEX on standard_bucket0(`index_name_0`) USING GSI 2022-02-03 12:12:25,147 - root - INFO - RUN QUERY BUILD INDEX on standard_bucket0(`index_name_0`) USING GSI 2022-02-03 12:12:25,152 - root - INFO - query params : statement=BUILD+INDEX+on+standard_bucket0%28%60index_name_0%60%29+USING+GSI 2022-02-03 12:12:25,224 - root - INFO - TOTAL ELAPSED TIME: 68.826392ms 2022-02-03 12:12:25,230 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-02-03 12:12:25,235 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-02-03 12:12:25,247 - root - INFO - TOTAL ELAPSED TIME: 8.157911ms 2022-02-03 12:12:26,257 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-02-03 12:12:26,262 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-02-03 12:12:26,289 - root - INFO - TOTAL ELAPSED TIME: 22.859127ms 2022-02-03 12:12:27,297 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-02-03 12:12:27,303 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-02-03 12:12:27,326 - root - INFO - TOTAL ELAPSED TIME: 18.400972ms 2022-02-03 12:12:28,337 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-02-03 12:12:28,342 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-02-03 12:12:28,360 - root - INFO - TOTAL ELAPSED TIME: 13.533758ms 2022-02-03 12:12:29,367 - root - INFO - RUN QUERY SELECT * FROM system:indexes where name = 'index_name_0' 2022-02-03 12:12:29,372 - root - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27index_name_0%27 2022-02-03 12:12:29,387 - root - INFO - TOTAL ELAPSED TIME: 12.32683ms 2022-02-03 12:12:30,395 - root - INFO - === BEFORE: [] === 2022-02-03 12:12:30,401 - root - INFO - RUN QUERY PREPARE prepare_employee95868e0d08c24f20bfbdf107c91a0a00job_titlestandard_bucket0 FROM SELECT * FROM standard_bucket0 where job_title IS NOT NULL ORDER BY job_title,_id 2022-02-03 12:12:30,406 - root - INFO - query params : statement=PREPARE+prepare_employee95868e0d08c24f20bfbdf107c91a0a00job_titlestandard_bucket0+FROM+SELECT+%2A+FROM+standard_bucket0+where+job_title+IS+NOT+NULL+ORDER+BY+job_title%2C_id 2022-02-03 12:12:30,411 - root - INFO - TOTAL ELAPSED TIME: 2.531953ms 2022-02-03 12:12:30,416 - root - INFO - RUN QUERY PREPARE prepare_employee95868e0d08c24f20bfbdf107c91a0a00join_yrstandard_bucket0 FROM SELECT * FROM standard_bucket0 WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id 2022-02-03 12:12:30,424 - root - INFO - query params : statement=PREPARE+prepare_employee95868e0d08c24f20bfbdf107c91a0a00join_yrstandard_bucket0+FROM+SELECT+%2A+FROM+standard_bucket0+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+ 2022-02-03 12:12:30,429 - root - INFO - TOTAL ELAPSED TIME: 2.03682ms 2022-02-03 12:12:30,441 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:12:30,452 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:12:30,518 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:12:30,603 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:30,856 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:30,907 - root - INFO - 10.112.205.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:12:31,016 - root - INFO - Running systemd stop command on this server 2022-02-03 12:12:31,016 - root - INFO - running command.raw on 10.112.205.101: systemctl stop couchbase-server.service 2022-02-03 12:12:32,687 - root - INFO - command executed successfully with root 2022-02-03 12:12:32,690 - root - INFO - SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-03 12:12:32,748 - root - INFO - SSH Connected to 10.112.205.102 as root 2022-02-03 12:12:32,877 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:33,152 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:33,218 - root - INFO - 10.112.205.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:12:33,288 - root - INFO - Running systemd stop command on this server 2022-02-03 12:12:33,289 - root - INFO - running command.raw on 10.112.205.102: systemctl stop couchbase-server.service 2022-02-03 12:12:34,803 - root - INFO - command executed successfully with root 2022-02-03 12:12:34,807 - root - INFO - SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-03 12:12:34,865 - root - INFO - SSH Connected to 10.112.205.103 as root 2022-02-03 12:12:34,945 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:35,201 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:35,284 - root - INFO - 10.112.205.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:12:35,355 - root - INFO - Running systemd stop command on this server 2022-02-03 12:12:35,356 - root - INFO - running command.raw on 10.112.205.103: systemctl stop couchbase-server.service 2022-02-03 12:12:36,852 - root - INFO - command executed successfully with root 2022-02-03 12:12:36,855 - root - INFO - SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-03 12:12:36,911 - root - INFO - SSH Connected to 10.112.205.104 as root 2022-02-03 12:12:36,994 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:37,247 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:37,299 - root - INFO - 10.112.205.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:12:37,366 - root - INFO - Running systemd stop command on this server 2022-02-03 12:12:37,367 - root - INFO - running command.raw on 10.112.205.104: systemctl stop couchbase-server.service 2022-02-03 12:12:38,907 - root - INFO - command executed successfully with root 2022-02-03 12:12:38,910 - root - INFO - SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-03 12:12:38,968 - root - INFO - SSH Connected to 10.112.205.105 as root 2022-02-03 12:12:39,053 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:39,299 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:39,350 - root - INFO - 10.112.205.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:12:39,454 - root - INFO - Running systemd stop command on this server 2022-02-03 12:12:39,454 - root - INFO - running command.raw on 10.112.205.105: systemctl stop couchbase-server.service 2022-02-03 12:12:40,933 - root - INFO - command executed successfully with root 2022-02-03 12:12:40,937 - root - INFO - SSH Connecting to 10.112.205.106 with username:root, attempt#1 of 5 2022-02-03 12:12:40,992 - root - INFO - SSH Connected to 10.112.205.106 as root 2022-02-03 12:12:41,074 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:41,327 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:41,377 - root - INFO - 10.112.205.106 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:12:41,444 - root - INFO - Running systemd stop command on this server 2022-02-03 12:12:41,445 - root - INFO - running command.raw on 10.112.205.106: systemctl stop couchbase-server.service 2022-02-03 12:12:42,934 - root - INFO - command executed successfully with root 2022-02-03 12:12:42,935 - root - INFO - servers ['10.112.205.101', '10.112.205.102', '10.112.205.103', '10.112.205.104', '10.112.205.105', '10.112.205.106'] will be upgraded to 7.1.0-2074 version 2022-02-03 12:12:42,941 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:12:42,944 - root - INFO - SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-03 12:12:42,946 - root - INFO - SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-03 12:12:42,950 - root - INFO - SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-03 12:12:42,959 - root - INFO - SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-03 12:12:42,963 - root - INFO - SSH Connecting to 10.112.205.106 with username:root, attempt#1 of 5 2022-02-03 12:12:43,054 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:12:43,057 - root - INFO - SSH Connected to 10.112.205.102 as root 2022-02-03 12:12:43,062 - root - INFO - SSH Connected to 10.112.205.104 as root 2022-02-03 12:12:43,066 - root - INFO - SSH Connected to 10.112.205.103 as root 2022-02-03 12:12:43,074 - root - INFO - SSH Connected to 10.112.205.105 as root 2022-02-03 12:12:43,092 - root - INFO - SSH Connected to 10.112.205.106 as root 2022-02-03 12:12:43,231 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:43,240 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:43,254 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:43,255 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:43,261 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:43,262 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:12:43,539 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:43,542 - root - INFO - finding build 7.1.0-2074 for machine ip:10.112.205.101 port:8091 ssh_username:root 2022-02-03 12:12:43,543 - root - INFO - running command.raw on 10.112.205.101: /sbin/iptables -F 2022-02-03 12:12:43,547 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:43,549 - root - INFO - finding build 7.1.0-2074 for machine ip:10.112.205.102 port:8091 ssh_username:root 2022-02-03 12:12:43,550 - root - INFO - running command.raw on 10.112.205.102: /sbin/iptables -F 2022-02-03 12:12:43,572 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:43,574 - root - INFO - finding build 7.1.0-2074 for machine ip:10.112.205.105 port:8091 ssh_username:root 2022-02-03 12:12:43,575 - root - INFO - running command.raw on 10.112.205.105: /sbin/iptables -F 2022-02-03 12:12:43,588 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:43,592 - root - INFO - finding build 7.1.0-2074 for machine ip:10.112.205.103 port:8091 ssh_username:root 2022-02-03 12:12:43,596 - root - INFO - running command.raw on 10.112.205.103: /sbin/iptables -F 2022-02-03 12:12:43,600 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:43,603 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:12:43,605 - root - INFO - finding build 7.1.0-2074 for machine ip:10.112.205.106 port:8091 ssh_username:root 2022-02-03 12:12:43,606 - root - INFO - running command.raw on 10.112.205.106: /sbin/iptables -F 2022-02-03 12:12:43,608 - root - INFO - finding build 7.1.0-2074 for machine ip:10.112.205.104 port:8091 ssh_username:root 2022-02-03 12:12:43,609 - root - INFO - running command.raw on 10.112.205.104: /sbin/iptables -F 2022-02-03 12:12:43,613 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,615 - root - INFO - running command.raw on 10.112.205.101: /sbin/iptables -t nat -F 2022-02-03 12:12:43,626 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,629 - root - INFO - running command.raw on 10.112.205.102: /sbin/iptables -t nat -F 2022-02-03 12:12:43,649 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,652 - root - INFO - running command.raw on 10.112.205.105: /sbin/iptables -t nat -F 2022-02-03 12:12:43,662 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,663 - root - INFO - running command.raw on 10.112.205.103: /sbin/iptables -t nat -F 2022-02-03 12:12:43,672 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,676 - root - INFO - running command.raw on 10.112.205.106: /sbin/iptables -t nat -F 2022-02-03 12:12:43,681 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,682 - root - INFO - running command.raw on 10.112.205.104: /sbin/iptables -t nat -F 2022-02-03 12:12:43,685 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,686 - root - INFO - SSH Connecting to 10.112.205.101 with username:root, attempt#1 of 5 2022-02-03 12:12:43,708 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,709 - root - INFO - SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-03 12:12:43,730 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,731 - root - INFO - SSH Connecting to 10.112.205.105 with username:root, attempt#1 of 5 2022-02-03 12:12:43,745 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,747 - root - INFO - SSH Connecting to 10.112.205.103 with username:root, attempt#1 of 5 2022-02-03 12:12:43,754 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,755 - root - INFO - SSH Connecting to 10.112.205.106 with username:root, attempt#1 of 5 2022-02-03 12:12:43,759 - root - INFO - command executed successfully with root 2022-02-03 12:12:43,760 - root - INFO - SSH Connecting to 10.112.205.104 with username:root, attempt#1 of 5 2022-02-03 12:12:43,765 - root - INFO - SSH Connected to 10.112.205.101 as root 2022-02-03 12:12:43,797 - root - INFO - deliverable_type : rpm 2022-02-03 12:12:43,798 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm or /tmp/ 2022-02-03 12:12:43,800 - root - INFO - running command.raw on 10.112.205.101: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm 2022-02-03 12:12:43,826 - root - INFO - SSH Connected to 10.112.205.102 as root 2022-02-03 12:12:43,830 - root - INFO - SSH Connected to 10.112.205.105 as root 2022-02-03 12:12:43,856 - root - INFO - deliverable_type : rpm 2022-02-03 12:12:43,858 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm or /tmp/ 2022-02-03 12:12:43,863 - root - INFO - SSH Connected to 10.112.205.103 as root 2022-02-03 12:12:43,864 - root - INFO - deliverable_type : rpm 2022-02-03 12:12:43,864 - root - INFO - running command.raw on 10.112.205.102: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm 2022-02-03 12:12:43,872 - root - INFO - SSH Connected to 10.112.205.106 as root 2022-02-03 12:12:43,875 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm or /tmp/ 2022-02-03 12:12:43,878 - root - INFO - running command.raw on 10.112.205.105: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm 2022-02-03 12:12:43,880 - root - INFO - SSH Connected to 10.112.205.104 as root 2022-02-03 12:12:43,908 - root - INFO - deliverable_type : rpm 2022-02-03 12:12:43,915 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm or /tmp/ 2022-02-03 12:12:43,918 - root - INFO - running command.raw on 10.112.205.103: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm 2022-02-03 12:12:43,921 - root - INFO - deliverable_type : rpm 2022-02-03 12:12:43,924 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm or /tmp/ 2022-02-03 12:12:43,929 - root - INFO - running command.raw on 10.112.205.106: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm 2022-02-03 12:12:43,932 - root - INFO - deliverable_type : rpm 2022-02-03 12:12:43,934 - root - INFO - /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm or /tmp/ 2022-02-03 12:12:43,935 - root - INFO - running command.raw on 10.112.205.104: export CB_MASTER_PASSWORD=password; rpm -U /tmp/couchbase-server-enterprise-7.1.0-2074-centos7.x86_64.rpm 2022-02-03 12:13:34,987 - root - INFO - command executed successfully with root 2022-02-03 12:13:34,989 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-02-03 12:13:34,991 - root - INFO - running command.raw on 10.112.205.106: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-02-03 12:13:34,996 - root - INFO - command executed successfully with root 2022-02-03 12:13:34,997 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-02-03 12:13:35,000 - root - INFO - running command.raw on 10.112.205.105: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-02-03 12:13:35,064 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,067 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,071 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,073 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,145 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,149 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-02-03 12:13:35,150 - root - INFO - running command.raw on 10.112.205.102: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-02-03 12:13:35,176 - root - INFO - process beam.smp is running on 10.112.205.105: with pid 4007 2022-02-03 12:13:35,194 - root - INFO - process beam.smp is running on 10.112.205.106: with pid 3800 2022-02-03 12:13:35,222 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,225 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,237 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,239 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-02-03 12:13:35,241 - root - INFO - running command.raw on 10.112.205.103: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-02-03 12:13:35,305 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,309 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,329 - root - INFO - Running systemd command on this server 2022-02-03 12:13:35,331 - root - INFO - running command.raw on 10.112.205.105: systemctl stop couchbase-server.service 2022-02-03 12:13:35,351 - root - INFO - process beam.smp is running on 10.112.205.102: with pid 4102 2022-02-03 12:13:35,354 - root - INFO - Running systemd command on this server 2022-02-03 12:13:35,363 - root - INFO - running command.raw on 10.112.205.106: systemctl stop couchbase-server.service 2022-02-03 12:13:35,405 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,408 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,413 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,417 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,426 - root - INFO - process beam.smp is running on 10.112.205.103: with pid 3999 2022-02-03 12:13:35,448 - root - INFO - process beam.smp is running on 10.112.205.105: with pid 4007 2022-02-03 12:13:35,452 - root - INFO - upgrade 10.112.205.105 to version 7.1.0-2074 is completed 2022-02-03 12:13:35,485 - root - INFO - process beam.smp is running on 10.112.205.106: with pid 3800 2022-02-03 12:13:35,489 - root - INFO - upgrade 10.112.205.106 to version 7.1.0-2074 is completed 2022-02-03 12:13:35,531 - root - INFO - 10.112.205.105 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:13:35,555 - root - INFO - 10.112.205.106 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:13:35,573 - root - INFO - Running systemd command on this server 2022-02-03 12:13:35,576 - root - INFO - running command.raw on 10.112.205.102: systemctl stop couchbase-server.service 2022-02-03 12:13:35,628 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,631 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,665 - root - INFO - Running systemd command on this server 2022-02-03 12:13:35,669 - root - INFO - running command.raw on 10.112.205.103: systemctl stop couchbase-server.service 2022-02-03 12:13:35,703 - root - INFO - process beam.smp is running on 10.112.205.102: with pid 4102 2022-02-03 12:13:35,706 - root - INFO - upgrade 10.112.205.102 to version 7.1.0-2074 is completed 2022-02-03 12:13:35,713 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,715 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:35,742 - root - INFO - Running systemd start command on this server 2022-02-03 12:13:35,745 - root - INFO - running command.raw on 10.112.205.106: systemctl start couchbase-server.service 2022-02-03 12:13:35,757 - root - INFO - Running systemd start command on this server 2022-02-03 12:13:35,760 - root - INFO - running command.raw on 10.112.205.105: systemctl start couchbase-server.service 2022-02-03 12:13:35,779 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,783 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-02-03 12:13:35,796 - root - INFO - process beam.smp is running on 10.112.205.103: with pid 3999 2022-02-03 12:13:35,803 - root - INFO - upgrade 10.112.205.103 to version 7.1.0-2074 is completed 2022-02-03 12:13:35,812 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,814 - root - INFO - 10.112.205.102 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:13:35,814 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-02-03 12:13:35,867 - root - INFO - 10.112.205.103 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:13:35,927 - root - INFO - Running systemd start command on this server 2022-02-03 12:13:35,928 - root - INFO - running command.raw on 10.112.205.102: systemctl start couchbase-server.service 2022-02-03 12:13:35,958 - root - INFO - command executed successfully with root 2022-02-03 12:13:35,960 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-02-03 12:13:35,992 - root - INFO - Running systemd start command on this server 2022-02-03 12:13:35,993 - root - INFO - running command.raw on 10.112.205.103: systemctl start couchbase-server.service 2022-02-03 12:13:36,023 - root - INFO - command executed successfully with root 2022-02-03 12:13:36,026 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-02-03 12:13:44,356 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,359 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-02-03 12:13:44,361 - root - INFO - running command.raw on 10.112.205.104: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-02-03 12:13:44,419 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,421 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:44,442 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,448 - root - INFO - Found 'You have successfully installed Couchbase Server.' in output 2022-02-03 12:13:44,450 - root - INFO - running command.raw on 10.112.205.101: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server 2022-02-03 12:13:44,498 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,500 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:44,523 - root - INFO - process beam.smp is running on 10.112.205.104: with pid 4990 2022-02-03 12:13:44,628 - root - INFO - process beam.smp is running on 10.112.205.101: with pid 5040 2022-02-03 12:13:44,651 - root - INFO - Running systemd command on this server 2022-02-03 12:13:44,653 - root - INFO - running command.raw on 10.112.205.104: systemctl stop couchbase-server.service 2022-02-03 12:13:44,707 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,710 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:44,748 - root - INFO - Running systemd command on this server 2022-02-03 12:13:44,750 - root - INFO - running command.raw on 10.112.205.101: systemctl stop couchbase-server.service 2022-02-03 12:13:44,776 - root - INFO - Starting couchbase server 2022-02-03 12:13:44,814 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,818 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:44,843 - root - INFO - process beam.smp is running on 10.112.205.101: with pid 5040 2022-02-03 12:13:44,845 - root - INFO - upgrade 10.112.205.101 to version 7.1.0-2074 is completed 2022-02-03 12:13:44,877 - root - INFO - Running systemd command on this server 2022-02-03 12:13:44,879 - root - INFO - running command.raw on 10.112.205.104: systemctl start couchbase-server.service 2022-02-03 12:13:44,900 - root - INFO - command executed successfully with root 2022-02-03 12:13:44,904 - root - INFO - 10.112.205.104:sleep for 5 secs. waiting for couchbase server to come up ... 2022-02-03 12:13:44,907 - root - INFO - 10.112.205.101 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:13:45,002 - root - INFO - Running systemd start command on this server 2022-02-03 12:13:45,003 - root - INFO - running command.raw on 10.112.205.101: systemctl start couchbase-server.service 2022-02-03 12:13:45,020 - root - INFO - command executed successfully with root 2022-02-03 12:13:45,023 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-02-03 12:13:49,911 - root - INFO - running command.raw on 10.112.205.104: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2022-02-03 12:13:50,040 - root - INFO - command executed successfully with root 2022-02-03 12:13:50,041 - root - INFO - Couchbase server status: [' Process: 5021 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=1/FAILURE)'] 2022-02-03 12:13:50,042 - root - INFO - Checking for process beam.smp on linux 2022-02-03 12:13:50,077 - root - INFO - process beam.smp is running on 10.112.205.104: with pid 5033 2022-02-03 12:13:50,078 - root - INFO - upgrade 10.112.205.104 to version 7.1.0-2074 is completed 2022-02-03 12:13:50,167 - root - INFO - 10.112.205.104 **** The linux version file /opt/couchbase/ VERSION.txt exists 2022-02-03 12:13:50,387 - root - INFO - Running systemd start command on this server 2022-02-03 12:13:50,388 - root - INFO - running command.raw on 10.112.205.104: systemctl start couchbase-server.service 2022-02-03 12:13:50,416 - root - INFO - command executed successfully with root 2022-02-03 12:13:50,418 - root - INFO - sleep for 30 secs. wait for servers up ... 2022-02-03 12:14:05,797 - root - INFO - waiting for ns_server @ 10.112.205.106:8091 2022-02-03 12:14:05,798 - root - INFO - -->is_ns_server_running? 2022-02-03 12:14:05,802 - root - INFO - ns_server @ 10.112.205.106:8091 is running 2022-02-03 12:14:05,802 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:14:05,803 - root - INFO - settings/web params on 10.112.205.106:8091:port=8091&username=Administrator&password=password 2022-02-03 12:14:05,833 - root - INFO - waiting for ns_server @ 10.112.205.105:8091 2022-02-03 12:14:05,834 - root - INFO - -->is_ns_server_running? 2022-02-03 12:14:05,839 - root - INFO - ns_server @ 10.112.205.105:8091 is running 2022-02-03 12:14:05,839 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:14:05,840 - root - INFO - settings/web params on 10.112.205.105:8091:port=8091&username=Administrator&password=password 2022-02-03 12:14:05,859 - root - INFO - --> status:True 2022-02-03 12:14:05,860 - root - INFO - sleep for 15 secs. ... 2022-02-03 12:14:05,926 - root - INFO - --> status:True 2022-02-03 12:14:05,927 - root - INFO - sleep for 15 secs. ... 2022-02-03 12:14:06,034 - root - INFO - waiting for ns_server @ 10.112.205.102:8091 2022-02-03 12:14:06,035 - root - INFO - -->is_ns_server_running? 2022-02-03 12:14:06,049 - root - INFO - ns_server @ 10.112.205.102:8091 is running 2022-02-03 12:14:06,050 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:14:06,051 - root - INFO - settings/web params on 10.112.205.102:8091:port=8091&username=Administrator&password=password 2022-02-03 12:14:06,076 - root - INFO - waiting for ns_server @ 10.112.205.103:8091 2022-02-03 12:14:06,078 - root - INFO - -->is_ns_server_running? 2022-02-03 12:14:06,089 - root - INFO - ns_server @ 10.112.205.103:8091 is running 2022-02-03 12:14:06,090 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:14:06,091 - root - INFO - settings/web params on 10.112.205.103:8091:port=8091&username=Administrator&password=password 2022-02-03 12:14:06,193 - root - INFO - --> status:True 2022-02-03 12:14:06,194 - root - INFO - sleep for 15 secs. ... 2022-02-03 12:14:06,200 - root - INFO - --> status:True 2022-02-03 12:14:06,201 - root - INFO - sleep for 15 secs. ... 2022-02-03 12:14:15,040 - root - INFO - waiting for ns_server @ 10.112.205.101:8091 2022-02-03 12:14:15,041 - root - INFO - -->is_ns_server_running? 2022-02-03 12:14:15,046 - root - INFO - ns_server @ 10.112.205.101:8091 is running 2022-02-03 12:14:15,047 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:14:15,048 - root - INFO - settings/web params on 10.112.205.101:8091:port=8091&username=Administrator&password=password 2022-02-03 12:14:15,144 - root - INFO - --> status:True 2022-02-03 12:14:15,145 - root - INFO - sleep for 15 secs. ... 2022-02-03 12:14:20,432 - root - INFO - waiting for ns_server @ 10.112.205.104:8091 2022-02-03 12:14:20,433 - root - INFO - -->is_ns_server_running? 2022-02-03 12:14:20,437 - root - INFO - ns_server @ 10.112.205.104:8091 is running 2022-02-03 12:14:20,438 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-03 12:14:20,438 - root - INFO - settings/web params on 10.112.205.104:8091:port=8091&username=Administrator&password=password 2022-02-03 12:14:20,518 - root - INFO - --> status:True 2022-02-03 12:14:20,519 - root - INFO - sleep for 15 secs. ... 2022-02-03 12:14:20,862 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:14:20,932 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:14:21,200 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:14:21,206 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:14:30,157 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:14:35,531 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:14:45,535 - root - INFO - **** add built-in 'cbadminbucket' user to node 10.112.205.101 **** 2022-02-03 12:14:45,691 - root - INFO - sleep for 5 secs. ... 2022-02-03 12:14:50,697 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2022-02-03 12:14:50,714 - root - INFO - sleep for 20 secs. ... 2022-02-03 12:15:12,415 - root - INFO - create 20160 to standard_bucket0 documents... 2022-02-03 12:15:12,467 - root - INFO - creating direct client 10.112.205.101:11210 standard_bucket0 2022-02-03 12:15:12,550 - root - INFO - creating direct client 10.112.205.104:11210 standard_bucket0 2022-02-03 12:15:33,254 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:15:33,260 - root - INFO - Making a rest request api=http://10.112.205.102:9102/getIndexStatus verb=GET params= client_cert=None verify=False 2022-02-03 12:15:33,291 - root - INFO - Making a rest request api=http://10.112.205.103:9102/getIndexStatus verb=GET params= client_cert=None verify=False 2022-02-03 12:15:33,315 - root - INFO - FROM clause ===== is standard_bucket0 2022-02-03 12:15:33,317 - root - INFO - WHERE clause ===== is doc["job_title"] != "Sales" 2022-02-03 12:15:33,317 - root - INFO - UNNEST clause ===== is None 2022-02-03 12:15:33,319 - root - INFO - SELECT clause ===== is {"job_title" : doc["job_title"],} 2022-02-03 12:15:33,320 - root - INFO - -->select_clause:{"job_title" : doc["job_title"],}; where_clause= doc["job_title"] != "Sales" 2022-02-03 12:15:33,321 - root - INFO - -->where_clause= doc["job_title"] != "Sales" 2022-02-03 12:15:33,720 - root - INFO - RUN QUERY SELECT job_title FROM standard_bucket0 WHERE job_title != "Sales" 2022-02-03 12:15:33,724 - root - INFO - query params : statement=SELECT+job_title+FROM+standard_bucket0++WHERE++job_title+%21%3D+%22Sales%22+&scan_consistency=request_plus 2022-02-03 12:15:33,999 - root - INFO - TOTAL ELAPSED TIME: 269.445901ms 2022-02-03 12:15:34,000 - root - INFO - Analyzing Actual Result 2022-02-03 12:15:34,008 - root - INFO - Analyzing Expected Result 2022-02-03 12:15:37,064 - root - INFO - RUN QUERY EXECUTE prepare_employee95868e0d08c24f20bfbdf107c91a0a00job_titlestandard_bucket0 2022-02-03 12:15:37,068 - root - INFO - query params : statement=EXECUTE+prepare_employee95868e0d08c24f20bfbdf107c91a0a00job_titlestandard_bucket0 2022-02-03 12:15:37,085 - root - ERROR - POST http://10.112.205.105:8093/query?statement=EXECUTE+prepare_employee95868e0d08c24f20bfbdf107c91a0a00job_titlestandard_bucket0 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'{\n"requestID": "064505bf-986e-45c1-b34b-6289f9ad4fa6",\n"errors": [{"code":4040,"msg":"No such prepared statement: prepare_employee95868e0d08c24f20bfbdf107c91a0a00job_titlestandard_bucket0, context: unset"}],\n"status": "fatal",\n"metrics": {"elapsedTime": "13.96177ms","executionTime": "13.91647ms","resultCount": 0,"resultSize": 0,"serviceLoad": 25,"errorCount": 1}\n}\n' auth: Administrator:password 2022-02-03 12:15:37,103 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:15:37,109 - root - INFO - Making a rest request api=http://10.112.205.102:9102/settings verb=POST params={"indexer.numPartitions": 2} client_cert=None verify=False 2022-02-03 12:15:37,125 - root - INFO - {'indexer.numPartitions': 2} set 2022-02-03 12:15:37,136 - root - INFO - RUN QUERY CREATE INDEX partitioned_idx1 ON standard_bucket0(name, age, join_yr) partition by hash(name, age, join_yr) USING GSI; 2022-02-03 12:15:37,148 - root - INFO - query params : statement=CREATE+INDEX+partitioned_idx1+ON+standard_bucket0%28name%2C+age%2C+join_yr%29+partition+by+hash%28name%2C+age%2C+join_yr%29+USING+GSI%3B 2022-02-03 12:15:40,410 - root - INFO - TOTAL ELAPSED TIME: 3.256799896s 2022-02-03 12:15:40,417 - root - INFO - RUN QUERY CREATE INDEX non_partitioned_idx1 ON standard_bucket0(name, age, join_yr) USING GSI; 2022-02-03 12:15:40,424 - root - INFO - query params : statement=CREATE+INDEX+non_partitioned_idx1+ON+standard_bucket0%28name%2C+age%2C+join_yr%29+USING+GSI%3B 2022-02-03 12:15:43,693 - root - INFO - TOTAL ELAPSED TIME: 3.266095988s 2022-02-03 12:15:43,700 - root - INFO - RUN QUERY select name, age, join_yr from standard_bucket0 USE INDEX (partitioned_idx1) where name='Kala' 2022-02-03 12:15:43,706 - root - INFO - query params : statement=select+name%2C+age%2C+join_yr+from+standard_bucket0+USE+INDEX+%28partitioned_idx1%29+where+name%3D%27Kala%27 2022-02-03 12:15:43,712 - root - INFO - TOTAL ELAPSED TIME: 3.097976ms 2022-02-03 12:15:43,717 - root - INFO - RUN QUERY select name, age, join_yr from standard_bucket0 USE INDEX (non_partitioned_idx1) where name='Kala' 2022-02-03 12:15:43,723 - root - INFO - query params : statement=select+name%2C+age%2C+join_yr+from+standard_bucket0+USE+INDEX+%28non_partitioned_idx1%29+where+name%3D%27Kala%27 2022-02-03 12:15:43,729 - root - INFO - TOTAL ELAPSED TIME: 2.229127ms 2022-02-03 12:15:43,735 - root - INFO - RUN QUERY select name, age, join_yr from standard_bucket0 USE INDEX (partitioned_idx1) where name is not missing AND age=50 offset 0 limit 10 2022-02-03 12:15:43,739 - root - INFO - query params : statement=select+name%2C+age%2C+join_yr+from+standard_bucket0+USE+INDEX+%28partitioned_idx1%29+where+name+is+not+missing+AND+age%3D50+offset+0+limit+10 2022-02-03 12:15:43,774 - root - INFO - TOTAL ELAPSED TIME: 32.085739ms 2022-02-03 12:15:43,779 - root - INFO - RUN QUERY select name, age, join_yr from standard_bucket0 USE INDEX (non_partitioned_idx1) where name is not missing AND age=50 offset 0 limit 10 2022-02-03 12:15:43,785 - root - INFO - query params : statement=select+name%2C+age%2C+join_yr+from+standard_bucket0+USE+INDEX+%28non_partitioned_idx1%29+where+name+is+not+missing+AND+age%3D50+offset+0+limit+10 2022-02-03 12:15:43,802 - root - INFO - TOTAL ELAPSED TIME: 13.513959ms 2022-02-03 12:15:43,808 - root - INFO - RUN QUERY select count(name), age from standard_bucket0 USE INDEX (partitioned_idx1) where name is not missing group by age 2022-02-03 12:15:43,813 - root - INFO - query params : statement=select+count%28name%29%2C+age+from+standard_bucket0+USE+INDEX+%28partitioned_idx1%29+where+name+is+not+missing+group+by+age 2022-02-03 12:15:43,832 - root - INFO - TOTAL ELAPSED TIME: 15.107163ms 2022-02-03 12:15:43,837 - root - INFO - RUN QUERY select count(name), age from standard_bucket0 USE INDEX (non_partitioned_idx1) where name is not missing group by age 2022-02-03 12:15:43,844 - root - INFO - query params : statement=select+count%28name%29%2C+age+from+standard_bucket0+USE+INDEX+%28non_partitioned_idx1%29+where+name+is+not+missing+group+by+age 2022-02-03 12:15:43,861 - root - INFO - TOTAL ELAPSED TIME: 13.508444ms 2022-02-03 12:15:43,867 - root - INFO - RUN QUERY select name, age, join_yr from standard_bucket0 USE INDEX (partitioned_idx1) where name is not missing AND age > 50 2022-02-03 12:15:43,873 - root - INFO - query params : statement=select+name%2C+age%2C+join_yr+from+standard_bucket0+USE+INDEX+%28partitioned_idx1%29+where+name+is+not+missing+AND+age+%3E+50 2022-02-03 12:15:43,895 - root - INFO - TOTAL ELAPSED TIME: 18.41089ms 2022-02-03 12:15:43,901 - root - INFO - RUN QUERY select name, age, join_yr from standard_bucket0 USE INDEX (non_partitioned_idx1) where name is not missing AND age > 50 2022-02-03 12:15:43,906 - root - INFO - query params : statement=select+name%2C+age%2C+join_yr+from+standard_bucket0+USE+INDEX+%28non_partitioned_idx1%29+where+name+is+not+missing+AND+age+%3E+50 2022-02-03 12:15:43,924 - root - INFO - TOTAL ELAPSED TIME: 13.638709ms 2022-02-03 12:15:43,942 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:15:43,992 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes with params: name=test_scope_1 2022-02-03 12:15:43,993 - root - INFO - Scope created standard_bucket0->test_scope_1 {'uid': '1'} 2022-02-03 12:15:44,121 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_1/collections with params: name=test_collection_1 2022-02-03 12:15:44,122 - root - INFO - Collection created standard_bucket0->test_scope_1->test_collection_1 manifest:{'uid': '2'} 2022-02-03 12:15:44,253 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_1/collections with params: name=test_collection_2 2022-02-03 12:15:44,255 - root - INFO - Collection created standard_bucket0->test_scope_1->test_collection_2 manifest:{'uid': '3'} 2022-02-03 12:15:44,350 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_1/collections with params: name=test_collection_3 2022-02-03 12:15:44,351 - root - INFO - Collection created standard_bucket0->test_scope_1->test_collection_3 manifest:{'uid': '4'} 2022-02-03 12:15:44,474 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes with params: name=test_scope_2 2022-02-03 12:15:44,475 - root - INFO - Scope created standard_bucket0->test_scope_2 {'uid': '5'} 2022-02-03 12:15:44,620 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_2/collections with params: name=test_collection_1 2022-02-03 12:15:44,621 - root - INFO - Collection created standard_bucket0->test_scope_2->test_collection_1 manifest:{'uid': '6'} 2022-02-03 12:15:44,738 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_2/collections with params: name=test_collection_2 2022-02-03 12:15:44,739 - root - INFO - Collection created standard_bucket0->test_scope_2->test_collection_2 manifest:{'uid': '7'} 2022-02-03 12:15:44,841 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_2/collections with params: name=test_collection_3 2022-02-03 12:15:44,842 - root - INFO - Collection created standard_bucket0->test_scope_2->test_collection_3 manifest:{'uid': '8'} 2022-02-03 12:15:44,966 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes with params: name=test_scope_3 2022-02-03 12:15:44,968 - root - INFO - Scope created standard_bucket0->test_scope_3 {'uid': '9'} 2022-02-03 12:15:45,105 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_3/collections with params: name=test_collection_1 2022-02-03 12:15:45,107 - root - INFO - Collection created standard_bucket0->test_scope_3->test_collection_1 manifest:{'uid': 'a'} 2022-02-03 12:15:45,229 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_3/collections with params: name=test_collection_2 2022-02-03 12:15:45,230 - root - INFO - Collection created standard_bucket0->test_scope_3->test_collection_2 manifest:{'uid': 'b'} 2022-02-03 12:15:45,353 - root - INFO - http://10.112.205.101:8091/pools/default/buckets/standard_bucket0/scopes/test_scope_3/collections with params: name=test_collection_3 2022-02-03 12:15:45,354 - root - INFO - Collection created standard_bucket0->test_scope_3->test_collection_3 manifest:{'uid': 'c'} 2022-02-03 12:15:45,356 - root - INFO - sleep for 10 secs. Allowing time after collection creation ... 2022-02-03 12:15:55,367 - root - INFO - Number of batches : 1 2022-02-03 12:15:55,368 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:16:05,371 - root - INFO - done 2022-02-03 12:16:05,595 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_1 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:16:08,901 - root - INFO - Number of batches : 1 2022-02-03 12:16:08,902 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:16:18,911 - root - INFO - done 2022-02-03 12:16:38,907 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_1 -c test_collection_2 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:16:41,378 - root - INFO - Number of batches : 1 2022-02-03 12:16:41,379 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:16:51,390 - root - INFO - done 2022-02-03 12:17:11,387 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_1 -c test_collection_3 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:17:14,119 - root - INFO - Number of batches : 1 2022-02-03 12:17:14,120 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:17:24,131 - root - INFO - done 2022-02-03 12:17:44,129 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_2 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:17:46,725 - root - INFO - Number of batches : 1 2022-02-03 12:17:46,726 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:17:56,736 - root - INFO - done 2022-02-03 12:18:16,734 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_2 -c test_collection_2 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:18:19,505 - root - INFO - Number of batches : 1 2022-02-03 12:18:19,507 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:18:29,514 - root - INFO - done 2022-02-03 12:18:49,514 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_2 -c test_collection_3 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:18:52,196 - root - INFO - Number of batches : 1 2022-02-03 12:18:52,197 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:19:02,208 - root - INFO - done 2022-02-03 12:19:22,205 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_3 -c test_collection_1 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:19:27,472 - root - INFO - Number of batches : 1 2022-02-03 12:19:27,473 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:19:37,477 - root - INFO - done 2022-02-03 12:19:57,477 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_3 -c test_collection_2 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:20:03,612 - root - INFO - Number of batches : 1 2022-02-03 12:20:03,614 - root - INFO - sleep for 10 secs. ... 2022-02-03 12:20:13,619 - root - INFO - done 2022-02-03 12:20:33,615 - root - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 10.112.205.101 -u Administrator -p password -b standard_bucket0 -s test_scope_3 -c test_collection_3 -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 999 -o True -sd False --secure False 2022-02-03 12:20:41,550 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:20:44,580 - root - INFO - list of index nodes in cluster: [ip:10.112.205.102 port:8091 ssh_username:root, ip:10.112.205.103 port:8091 ssh_username:root] 2022-02-03 12:20:44,589 - root - INFO - settings/autoFailover params : timeout=60&enabled=true&canAbortRebalance=true&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 2022-02-03 12:20:44,605 - root - INFO - sleep for 5 secs. ... 2022-02-03 12:20:49,614 - root - INFO - sleep for 5 secs. ... 2022-02-03 12:20:54,620 - root - INFO - Initializing NodeDownTimerTask 2022-02-03 12:20:55,437 - root - INFO - before failure time: Thu Feb 3 12:20:55 2022 2022-02-03 12:20:56,446 - root - INFO - SSH Connecting to 10.112.205.102 with username:root, attempt#1 of 5 2022-02-03 12:20:56,517 - root - INFO - SSH Connected to 10.112.205.102 as root 2022-02-03 12:20:56,607 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-03 12:20:56,866 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-03 12:20:56,939 - root - INFO - Running systemd command on this server 2022-02-03 12:20:56,939 - root - INFO - running command.raw on 10.112.205.102: systemctl stop couchbase-server.service 2022-02-03 12:20:57,434 - root - INFO - Starting execution of NodeDownTimerTask 2022-02-03 12:20:58,151 - root - INFO - Injected failure in 10.112.205.102 2022-02-03 12:20:59,135 - root - INFO - command executed successfully with root 2022-02-03 12:20:59,138 - root - INFO - Stopped the couchbase server on ip:10.112.205.102 port:8091 ssh_username:root 2022-02-03 12:20:59,140 - root - INFO - Start time = Thu Feb 3 12:20:57 2022 2022-02-03 12:26:17,811 - root - INFO - Latest logs from UI on 10.112.205.101: 2022-02-03 12:26:17,813 - root - ERROR - {'node': 'ns_1@10.112.205.101', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1643871131093, 'shortText': 'message', 'text': "Could not automatically fail over node ('ns_1@10.112.205.102') due to operation being unsafe for service index. Failing over nodes 10.112.205.102:9102(ce48e35bcf23d34ad6f868457ca6cf4b) would lose the following indexes/partitions: standard_bucket0._default._default.partitioned_idx1 2 1", 'serverTime': '2022-02-02T22:52:11.093Z'} 2022-02-03 12:26:17,814 - root - ERROR - {'node': 'ns_1@10.112.205.101', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1643871128094, 'shortText': 'message', 'text': "Could not automatically fail over node ('ns_1@10.112.205.102') due to operation being unsafe for service index. Failing over nodes 10.112.205.102:9102(ce48e35bcf23d34ad6f868457ca6cf4b) would lose the following indexes/partitions: standard_bucket0._default._default.partitioned_idx1 1 2", 'serverTime': '2022-02-02T22:52:08.094Z'} 2022-02-03 12:26:17,814 - root - ERROR - {'node': 'ns_1@10.112.205.101', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1643871067017, 'shortText': 'message', 'text': 'Enabled auto-failover with timeout 60 and max count 1', 'serverTime': '2022-02-02T22:51:07.017Z'} 2022-02-03 12:26:17,815 - root - ERROR - {'node': 'ns_1@10.112.205.101', 'type': 'info', 'code': 0, 'module': 'mb_master', 'tstamp': 1643871066953, 'shortText': 'message', 'text': "Haven't heard from a higher priority node or a master, so I'm taking over.", 'serverTime': '2022-02-02T22:51:06.953Z'} 2022-02-03 12:26:17,816 - root - ERROR - {'node': 'ns_1@10.112.205.105', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1643871058166, 'shortText': 'node down', 'text': "Node 'ns_1@10.112.205.105' saw that node 'ns_1@10.112.205.102' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-02-02T22:50:58.166Z'} 2022-02-03 12:26:17,817 - root - ERROR - {'node': 'ns_1@10.112.205.104', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1643871058138, 'shortText': 'node down', 'text': "Node 'ns_1@10.112.205.104' saw that node 'ns_1@10.112.205.102' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-02-02T22:50:58.138Z'} 2022-02-03 12:26:17,817 - root - ERROR - {'node': 'ns_1@10.112.205.101', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1643871058126, 'shortText': 'node down', 'text': "Node 'ns_1@10.112.205.101' saw that node 'ns_1@10.112.205.102' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-02-02T22:50:58.126Z'} 2022-02-03 12:26:17,818 - root - ERROR - {'node': 'ns_1@10.112.205.103', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1643871058126, 'shortText': 'node down', 'text': "Node 'ns_1@10.112.205.103' saw that node 'ns_1@10.112.205.102' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-02-02T22:50:58.126Z'} 2022-02-03 12:26:17,819 - root - ERROR - {'node': 'ns_1@10.112.205.102', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1643871044619, 'shortText': 'message', 'text': 'Updating auto-failover timeout to 60', 'serverTime': '2022-02-02T22:50:44.619Z'} 2022-02-03 12:26:17,819 - root - ERROR - {'node': 'ns_1@10.112.205.102', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1643870638123, 'shortText': 'message', 'text': 'Enabled auto-failover with timeout 120 and max count 1', 'serverTime': '2022-02-02T22:43:58.123Z'} 2022-02-03 12:26:17,820 - root - ERROR - Autofailover of node 10.112.205.102 was not initiated after the expected timeout period of 60 [, , , , , , , ] Thu Feb 3 12:26:17 2022 [, , , , , , , , , , , , ]