Started by user Balakumaran Gopal Rebuilds build #4 Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on magma-sc1205-32g-20c (magma_slave) in workspace /data/workspace/temp_rebalance_non_root [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... [WS-CLEANUP] Done No credentials specified Cloning the remote Git repository Cloning repository https://github.com/couchbase/testrunner/ > /usr/bin/git init /data/workspace/temp_rebalance_non_root # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner/ +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner/ # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner/ > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner/ +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 21827bd5d957cb31b048604e084de7ab0b10423a (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 21827bd5d957cb31b048604e084de7ab0b10423a # timeout=10 Commit message: "Validating RBAC user roles post upgrade" > /usr/bin/git rev-list --no-walk 21827bd5d957cb31b048604e084de7ab0b10423a # timeout=10 [temp_rebalance_non_root] $ /bin/sh -xe /tmp/jenkins6871471374192272055.sh + echo '[global] username:nonroot password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 [_1] ip:172.23.104.172 services:kv,index,n1ql,fts [_2] ip:172.23.105.127 services:kv,index,n1ql,fts [_3] ip:172.23.106.52 services:kv,index,n1ql,fts [_4] ip:172.23.99.128 services:kv,index,n1ql,fts' + python3 testrunner.py -i /tmp/win10_bucket_ops_ini_non_root.ini -p bucket_storage=couchstore,get-cbcollect-info=True,nodes_init=4 -t failover.failovertests.FailoverTests.test_failover_stop_server,replicas=1,graceful=False,num_failed_nodes=1,numViews=1,withViewsOps=True,createIndexesDuringFailover=True,items=10000,failoverMaster=True,bucket_storage=couchstore INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: failover.failovertests.FailoverTests.test_failover_stop_server,replicas=1,graceful=False,num_failed_nodes=1,numViews=1,withViewsOps=True,createIndexesDuringFailover=True,items=10000,failoverMaster=True,bucket_storage=couchstore INFO:__main__:Logs folder: /data/workspace/temp_rebalance_non_root/logs/testrunner-22-Feb-22_01-21-18/test_1 *** TestRunner *** {'bucket_storage': 'couchstore', 'cluster_name': 'win10_bucket_ops_ini_non_root', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10_bucket_ops_ini_non_root.ini', 'nodes_init': '4', 'num_nodes': 4} Logs will be stored at /data/workspace/temp_rebalance_non_root/logs/testrunner-22-Feb-22_01-21-18/test_1 ./testrunner -i /tmp/win10_bucket_ops_ini_non_root.ini -p bucket_storage=couchstore,get-cbcollect-info=True,nodes_init=4 -t failover.failovertests.FailoverTests.test_failover_stop_server,replicas=1,graceful=False,num_failed_nodes=1,numViews=1,withViewsOps=True,createIndexesDuringFailover=True,items=10000,failoverMaster=True,bucket_storage=couchstore Test Input params: {'replicas': '1', 'graceful': 'False', 'num_failed_nodes': '1', 'numViews': '1', 'withViewsOps': 'True', 'createIndexesDuringFailover': 'True', 'items': '10000', 'failoverMaster': 'True', 'bucket_storage': 'couchstore', 'ini': '/tmp/win10_bucket_ops_ini_non_root.ini', 'cluster_name': 'win10_bucket_ops_ini_non_root', 'get-cbcollect-info': 'True', 'nodes_init': '4', 'num_nodes': 4, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/temp_rebalance_non_root/logs/testrunner-22-Feb-22_01-21-18/test_1'} Run before suite setup for failover.failovertests.FailoverTests.test_failover_stop_server suite_setUp (unittest.loader._FailedTest) ... ERROR ====================================================================== ERROR: suite_setUp (unittest.loader._FailedTest) ---------------------------------------------------------------------- AttributeError: type object 'FailoverTests' has no attribute 'suite_setUp' ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) test_failover_stop_server (failover.failovertests.FailoverTests) ... -->before_suite_name:failover.failovertests.FailoverTests.suite_setUp,suite: ]> -->result: 2022-02-22 01:21:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2347-enterprise 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_failover_stop_server============== 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [failoverbasetests.tearDown] ============== tearDown was started for test #1 test_failover_stop_server ============== 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:21:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.172: with pid 80995 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as root 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: /sbin/iptables -F 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: /sbin/iptables -t nat -F 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:21:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: killall -SIGCONT memcached 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:21:21 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: killall -SIGCONT beam 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:21:21 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.105.127: with pid 24334 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as root 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:21:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: /sbin/iptables -F 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: /sbin/iptables -t nat -F 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: killall -SIGCONT memcached 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:21:22 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: killall -SIGCONT beam 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:21:22 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:21:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.52: with pid 28221 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as root 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: /sbin/iptables -F 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: /sbin/iptables -t nat -F 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:21:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: killall -SIGCONT memcached 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:21:24 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: killall -SIGCONT beam 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:21:24 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:24 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.99.128: with pid 12581 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:root, attempt#1 of 5 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as root 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: /sbin/iptables -F 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: /sbin/iptables -t nat -F 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: killall -SIGCONT memcached 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:21:25 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: killall -SIGCONT beam 2022-02-22 01:21:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:21:25 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.172, nothing to delete 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.99.128, nothing to delete 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [failoverbasetests.tearDown] ============== tearDown was finished for test #1 test_failover_stop_server ============== 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.270433905146317, 'mem_free': 3246149632, 'mem_total': 3947372544, 'swap_mem_used': 52854784, 'swap_mem_total': 3758092288} 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_failover_stop_server ============== 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.172, nothing to delete 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.99.128, nothing to delete 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-02-22 01:21:48 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.172:8091 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.172:8091 is running 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.127:8091 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.127:8091 is running 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.52:8091 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.52:8091 is running 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.99.128:8091 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.99.128:8091 is running 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_failover_stop_server ============== 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-02-22 01:21:48 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '903', 'memoryTotal': 3947372544, 'memoryFree': 3242389504, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2347-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.104.172 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.104.172 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1249 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-02-22 01:21:49 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '875', 'memoryTotal': 4201627648, 'memoryFree': 3526529024, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2347-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '904', 'memoryTotal': 3947372544, 'memoryFree': 3240255488, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2347-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2017 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.99.128 port:8091 ssh_username:nonroot, nodes/self 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '864', 'memoryTotal': 3961217024, 'memoryFree': 3357126656, 'mcdMemoryReserved': 3022, 'mcdMemoryAllocated': 3022, 'status': 'healthy', 'hostname': '172.23.99.128:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2347-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1256, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.99.128', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3777, 'curr_items': 0} 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.99.128:8091:port=8091&username=Administrator&password=password 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:21:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.99.128:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.99.128:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-02-22 01:21:53 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-02-22 01:21:53 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** 2022-02-22 01:21:53 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-02-22 01:21:53 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-02-22 01:21:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-02-22 01:21:58 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-02-22 01:21:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-02-22 01:21:59 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.127:8091 to cluster 2022-02-22 01:21:59 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 2022-02-22 01:22:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-02-22 01:22:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-22 01:22:24 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.52:8091 to cluster 2022-02-22 01:22:24 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 2022-02-22 01:22:34 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-02-22 01:22:34 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-22 01:22:52 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.99.128:8091 to cluster 2022-02-22 01:22:52 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.99.128:18091 to this cluster @172.23.104.172:8091 2022-02-22 01:23:02 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-02-22 01:23:02 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-02-22 01:23:17 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.105.127 not part of cluster inactiveAdded 2022-02-22 01:23:17 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.52 not part of cluster inactiveAdded 2022-02-22 01:23:17 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.99.128 not part of cluster inactiveAdded 2022-02-22 01:23:17 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.127,ns_1@172.23.106.52,ns_1@172.23.99.128', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-02-22 01:23:17 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-02-22 01:23:27 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-02-22 01:23:27 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.022041082382202 sec 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-02-22 01:23:27 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.172 2022-02-22 01:23:28 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-02-22 01:23:28 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.04 seconds to create bucket default 2022-02-22 01:23:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.172 to accept set ops 2022-02-22 01:23:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:23:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:23:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:23:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:23:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:23:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:23:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:23:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:23:32 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1249 2022-02-22 01:23:32 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_failover_stop_server ============== 2022-02-22 01:23:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:23:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:23:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:23:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:23:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:23:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:23:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:23:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:23:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:23:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:23:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:23:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:23:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:23:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:23:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:23:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 17.60780287474333, 'mem_free': 3137675264, 'mem_total': 3947372544, 'swap_mem_used': 52854784, 'swap_mem_total': 3758092288} 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.392694063926941, 'mem_free': 3456778240, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 11.82823347904346, 'mem_free': 3177664512, 'mem_total': 3947372544, 'swap_mem_used': 66347008, 'swap_mem_total': 3758092288} 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.99.128:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 7.359635811836116, 'mem_free': 3288305664, 'mem_total': 3961217024, 'swap_mem_used': 62943232, 'swap_mem_total': 3758092288} 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 145.3236048221588 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failoverbasetests.setUp] ============== FailoverBaseTest setup was finished for test #1 test_failover_stop_server ============== 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failovertests.common_test_body] Picking node 172.23.105.127 as reference node for test case 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failovertests.print_test_params] num_replicas : 1 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failovertests.print_test_params] recoveryType : ['delta'] 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failovertests.print_test_params] failover_reason : stop_server 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failovertests.print_test_params] num_failed_nodes : 1 2022-02-22 01:23:43 | INFO | MainProcess | test_thread | [failovertests.print_test_params] picking server : ip:172.23.105.127 port:8091 ssh_username:nonroot as the master 2022-02-22 01:23:44 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2347-enterprise', '7.1.0-2347-enterprise', '7.1.0-2347-enterprise', '7.1.0-2347-enterprise'] 2022-02-22 01:23:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 120 secs. ... 2022-02-22 01:25:44 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:25:44 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:25:44 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:25:44 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:25:45 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:25:45 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:25:45 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:25:45 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-02-22 01:25:45 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:25:45 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:25:45 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:25:45 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.127:11210 default 2022-02-22 01:25:45 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:25:45 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:25:45 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:25:45 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-02-22 01:25:45 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:25:45 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:25:45 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:25:45 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.99.128:11210 default 2022-02-22 01:25:46 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2347-enterprise', '7.1.0-2347-enterprise', '7.1.0-2347-enterprise', '7.1.0-2347-enterprise'] 2022-02-22 01:25:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot, ip:172.23.99.128 port:8091 ssh_username:nonroot] 2022-02-22 01:25:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:47 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:25:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:47 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:25:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:48 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket 2022-02-22 01:25:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:25:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-02-22 01:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:53 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket 2022-02-22 01:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-02-22 01:25:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:57 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket 2022-02-22 01:25:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:25:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.99.128:8091',default bucket 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:25:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:25:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:25:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-02-22 01:26:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:00 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.99.128:8091',default bucket 2022-02-22 01:26:00 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2347-enterprise', '7.1.0-2347-enterprise', '7.1.0-2347-enterprise', '7.1.0-2347-enterprise'] 2022-02-22 01:26:00 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot, ip:172.23.99.128 port:8091 ssh_username:nonroot] 2022-02-22 01:26:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:26:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:26:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.105.127:8091''172.23.106.52:8091''172.23.99.128:8091',default bucket 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:26:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:26:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:26:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:26:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.105.127:8091''172.23.106.52:8091''172.23.99.128:8091',default bucket 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:26:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:10 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:26:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:26:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:26:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.105.127:8091''172.23.106.52:8091''172.23.99.128:8091',default bucket 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:26:13 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:14 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:14 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:14 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:26:14 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:26:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:26:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-02-22 01:26:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:17 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.105.127:8091''172.23.106.52:8091''172.23.99.128:8091',default bucket 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.127:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password 2022-02-22 01:26:18 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] PUT http://172.23.105.127:8092//default/_design/dev_ddoc1 body: {"views": {"default_view0": {"map": "function (doc) {\n emit(doc._id, doc);\n}"}}} headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: {bad_return_value, {undef, [{snappy,compress, [[1, [<<1,16,0,0,27>>, <<"_design/dev_ddoc1">>, <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,0,34,0,0,0,0,0,1,0,202,194, 127,133>>]]], []}, {couch_compress,compress,1, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl"}, {line,20}]}, {couch_btree,'-write_node/3-lc$^0/1-0-',5, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,514}]}, {couch_btree,write_node,3, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,525}]}, {couch_btree,modify_node,8, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,418}]}, {couch_btree,query_modify_raw,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,261}]}, {couch_db_updater,update_docs_int,4, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,601}]}, {couch_db_updater,handle_info,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,317}]}]}} b'{"error":"{bad_return_value,\\n {undef,\\n [{snappy,compress,\\n [[1,\\n [<<1,16,0,0,27>>,\\n <<\\"_design/dev_ddoc1\\">>,\\n <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,0,34,0,0,0,0,0,1,0,202,194,\\n 127,133>>]]],\\n []},\\n {couch_compress,compress,1,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl\\"},\\n {line,20}]},\\n {couch_btree,\'-write_node/3-lc$^0/1-0-\',5,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,514}]},\\n {couch_btree,write_node,3,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,525}]},\\n {couch_btree,modify_node,8,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,418}]},\\n {couch_btree,query_modify_raw,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,261}]},\\n {couch_db_updater,update_docs_int,4,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,601}]},\\n {couch_db_updater,handle_info,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,317}]}]}}","reason":"{gen_server,call,\\n [\'capi_ddoc_manager-default\',\\n {interactive_update,{doc,<<\\"_design/dev_ddoc1\\">>,\\n {0,<<>>},\\n <<\\"{\\\\\\"views\\\\\\":{\\\\\\"default_view0\\\\\\":{\\\\\\"map\\\\\\":\\\\\\"function (doc) {\\\\\\\\n emit(doc._id, doc);\\\\\\\\n}\\\\\\"}}}\\">>,\\n 0,false,[]}},\\n infinity]}"}\n' auth: Administrator:password [, , , , , , , , , , ] 2022-02-22 01:26:18 | INFO | MainProcess | test_thread | [failoverbasetests.tearDown] ============== tearDown was started for test #1 test_failover_stop_server ============== [, , , , , ] Tue Feb 22 01:26:18 2022 2022-02-22 01:26:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.127:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password 2022-02-22 01:26:18 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] PUT http://172.23.105.127:8092//default/_design/dev_ddoc1 body: {"views": {"default_view1": {"map": "function (doc) {\n emit(doc._id, doc);\n}"}}} headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: {bad_return_value, {undef, [{snappy,compress, [[1, [<<1,16,0,0,27>>, <<"_design/dev_ddoc1">>, <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,0,122,0,0,0,0,0,1,0,203,137, 99,235>>]]], []}, {couch_compress,compress,1, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl"}, {line,20}]}, {couch_btree,'-write_node/3-lc$^0/1-0-',5, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,514}]}, {couch_btree,write_node,3, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,525}]}, {couch_btree,modify_node,8, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,418}]}, {couch_btree,query_modify_raw,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,261}]}, {couch_db_updater,update_docs_int,4, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,601}]}, {couch_db_updater,handle_info,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,317}]}]}} b'{"error":"{bad_return_value,\\n {undef,\\n [{snappy,compress,\\n [[1,\\n [<<1,16,0,0,27>>,\\n <<\\"_design/dev_ddoc1\\">>,\\n <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,0,122,0,0,0,0,0,1,0,203,137,\\n 99,235>>]]],\\n []},\\n {couch_compress,compress,1,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl\\"},\\n {line,20}]},\\n {couch_btree,\'-write_node/3-lc$^0/1-0-\',5,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,514}]},\\n {couch_btree,write_node,3,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,525}]},\\n {couch_btree,modify_node,8,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,418}]},\\n {couch_btree,query_modify_raw,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,261}]},\\n {couch_db_updater,update_docs_int,4,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,601}]},\\n {couch_db_updater,handle_info,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,317}]}]}}","reason":"{gen_server,call,\\n [\'capi_ddoc_manager-default\',\\n {interactive_update,{doc,<<\\"_design/dev_ddoc1\\">>,\\n {0,<<>>},\\n <<\\"{\\\\\\"views\\\\\\":{\\\\\\"default_view1\\\\\\":{\\\\\\"map\\\\\\":\\\\\\"function (doc) {\\\\\\\\n emit(doc._id, doc);\\\\\\\\n}\\\\\\"}}}\\">>,\\n 0,false,[]}},\\n infinity]}"}\n' auth: Administrator:password [, , , , , ] Tue Feb 22 01:26:18 2022 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.127:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password 2022-02-22 01:26:18 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] PUT http://172.23.105.127:8092//default/_design/dev_ddoc1 body: {"views": {"default_view2": {"map": "function (doc) {\n emit(doc._id, doc);\n}"}}} headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: {bad_return_value, {undef, [{snappy,compress, [[1, [<<1,16,0,0,27>>, <<"_design/dev_ddoc1">>, <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,0,210,0,0,0,0,0,1,0,125,146, 185,154>>]]], []}, {couch_compress,compress,1, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl"}, {line,20}]}, {couch_btree,'-write_node/3-lc$^0/1-0-',5, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,514}]}, {couch_btree,write_node,3, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,525}]}, {couch_btree,modify_node,8, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,418}]}, {couch_btree,query_modify_raw,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,261}]}, {couch_db_updater,update_docs_int,4, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,601}]}, {couch_db_updater,handle_info,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,317}]}]}} b'{"error":"{bad_return_value,\\n {undef,\\n [{snappy,compress,\\n [[1,\\n [<<1,16,0,0,27>>,\\n <<\\"_design/dev_ddoc1\\">>,\\n <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,0,210,0,0,0,0,0,1,0,125,146,\\n 185,154>>]]],\\n []},\\n {couch_compress,compress,1,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl\\"},\\n {line,20}]},\\n {couch_btree,\'-write_node/3-lc$^0/1-0-\',5,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,514}]},\\n {couch_btree,write_node,3,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,525}]},\\n {couch_btree,modify_node,8,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,418}]},\\n {couch_btree,query_modify_raw,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,261}]},\\n {couch_db_updater,update_docs_int,4,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,601}]},\\n {couch_db_updater,handle_info,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,317}]}]}}","reason":"{gen_server,call,\\n [\'capi_ddoc_manager-default\',\\n {interactive_update,{doc,<<\\"_design/dev_ddoc1\\">>,\\n {0,<<>>},\\n <<\\"{\\\\\\"views\\\\\\":{\\\\\\"default_view2\\\\\\":{\\\\\\"map\\\\\\":\\\\\\"function (doc) {\\\\\\\\n emit(doc._id, doc);\\\\\\\\n}\\\\\\"}}}\\">>,\\n 0,false,[]}},\\n infinity]}"}\n' auth: Administrator:password [, , , , , ] Tue Feb 22 01:26:18 2022 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.127:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password 2022-02-22 01:26:18 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] PUT http://172.23.105.127:8092//default/_design/dev_ddoc1 body: {"views": {"default_view3": {"map": "function (doc) {\n emit(doc._id, doc);\n}"}}} headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: noproc b'{"error":"noproc","reason":"{gen_server,call,\\n [\'capi_ddoc_manager-default\',\\n {interactive_update,{doc,<<\\"_design/dev_ddoc1\\">>,\\n {0,<<>>},\\n <<\\"{\\\\\\"views\\\\\\":{\\\\\\"default_view3\\\\\\":{\\\\\\"map\\\\\\":\\\\\\"function (doc) {\\\\\\\\n emit(doc._id, doc);\\\\\\\\n}\\\\\\"}}}\\">>,\\n 0,false,[]}},\\n infinity]}"}\n' auth: Administrator:password [, , , , , ] Tue Feb 22 01:26:18 2022 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.127:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password 2022-02-22 01:26:18 | INFO | MainProcess | Cluster_Thread | [rest_client.create_design_document] -->create_design_document 2022-02-22 01:26:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] PUT http://172.23.105.127:8092//default/_design/dev_ddoc1 body: {"views": {"default_view4": {"map": "function (doc) {\n emit(doc._id, doc);\n}"}}} headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: {bad_return_value, {undef, [{snappy,compress, [[1, [<<1,16,0,0,27>>, <<"_design/dev_ddoc1">>, <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,1,42,0,0,0,0,0,1,0,13,211, 114,174>>]]], []}, {couch_compress,compress,1, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl"}, {line,20}]}, {couch_btree,'-write_node/3-lc$^0/1-0-',5, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,514}]}, {couch_btree,write_node,3, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,525}]}, {couch_btree,modify_node,8, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,418}]}, {couch_btree,query_modify_raw,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl"}, {line,261}]}, {couch_db_updater,update_docs_int,4, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,601}]}, {couch_db_updater,handle_info,2, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl"}, {line,317}]}]}} b'{"error":"{bad_return_value,\\n {undef,\\n [{snappy,compress,\\n [[1,\\n [<<1,16,0,0,27>>,\\n <<\\"_design/dev_ddoc1\\">>,\\n <<0,0,0,0,0,1,0,0,0,88,0,0,0,0,1,42,0,0,0,0,0,1,0,13,211,\\n 114,174>>]]],\\n []},\\n {couch_compress,compress,1,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_compress.erl\\"},\\n {line,20}]},\\n {couch_btree,\'-write_node/3-lc$^0/1-0-\',5,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,514}]},\\n {couch_btree,write_node,3,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,525}]},\\n {couch_btree,modify_node,8,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,418}]},\\n {couch_btree,query_modify_raw,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_btree.erl\\"},\\n {line,261}]},\\n {couch_db_updater,update_docs_int,4,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,601}]},\\n {couch_db_updater,handle_info,2,\\n [{file,\\n \\"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couchdb/couch_db_updater.erl\\"},\\n {line,317}]}]}}","reason":"{gen_server,call,\\n [\'capi_ddoc_manager-default\',\\n {interactive_update,{doc,<<\\"_design/dev_ddoc1\\">>,\\n {0,<<>>},\\n <<\\"{\\\\\\"views\\\\\\":{\\\\\\"default_view4\\\\\\":{\\\\\\"map\\\\\\":\\\\\\"function (doc) {\\\\\\\\n emit(doc._id, doc);\\\\\\\\n}\\\\\\"}}}\\">>,\\n 0,false,[]}},\\n infinity]}"}\n' auth: Administrator:password [, , , , , ] Tue Feb 22 01:26:18 2022 2022-02-22 01:26:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:26:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.172: with pid 80995 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as root 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: /sbin/iptables -F 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: /sbin/iptables -t nat -F 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: killall -SIGCONT memcached 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:26:19 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: killall -SIGCONT beam 2022-02-22 01:26:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:26:19 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.105.127: with pid 24334 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as root 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:26:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: /sbin/iptables -F 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: /sbin/iptables -t nat -F 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: killall -SIGCONT memcached 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:26:21 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: killall -SIGCONT beam 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:26:21 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:26:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.52: with pid 28221 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as root 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: /sbin/iptables -F 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: /sbin/iptables -t nat -F 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:26:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: killall -SIGCONT memcached 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:26:23 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: killall -SIGCONT beam 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:26:23 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.99.128: with pid 12581 2022-02-22 01:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:root, attempt#1 of 5 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as root 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.disable_firewall] Non root or non sudo has no right to disable firewall, switching over to root 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: /sbin/iptables -F 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: /sbin/iptables -t nat -F 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.99.128 with username:nonroot, attempt#1 of 5 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.99.128 as nonroot 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.unpause_memcached] *** unpause memcached process *** 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: killall -SIGCONT memcached 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['memcached: no process found'] ... 2022-02-22 01:26:24 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] memcached: no process found 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.99.128: killall -SIGCONT beam 2022-02-22 01:26:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with nonroot but got an error ['beam: no process found'] ... 2022-02-22 01:26:24 | ERROR | MainProcess | test_thread | [remote_util.log_command_output] beam: no process found 2022-02-22 01:26:34 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.104.172 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.172 existing buckets : [] 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.104.172 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.99.128, nothing to delete 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.127,ns_1@172.23.106.52,ns_1@172.23.99.128', 'ejectedNodes': 'ns_1@172.23.105.127,ns_1@172.23.106.52,ns_1@172.23.99.128', 'user': 'Administrator', 'password': 'password'} 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-02-22 01:26:35 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-02-22 01:26:45 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2022-02-22 01:26:55 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2022-02-22 01:27:15 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2022-02-22 01:27:15 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... Build was aborted Aborted by Balakumaran Gopal Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: Finished: ABORTED