Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s23202 (P0 jython_slave PY3P0 vikas JP0) in workspace /data/workspace/temp_rebalance_even_win2 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository https://github.com/couchbase/testrunner/ > /usr/bin/git init /data/workspace/temp_rebalance_even_win2 # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner/ # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner/ > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision b2a06033c6019cce7a01a020005bbffc1ea0fb65 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f b2a06033c6019cce7a01a020005bbffc1ea0fb65 > /usr/bin/git rev-list b2a06033c6019cce7a01a020005bbffc1ea0fb65 # timeout=10 [temp_rebalance_even_win2] $ /bin/sh -xe /tmp/jenkins7367807187476328691.sh + echo '[global] username:Administrator password:Membase123 port:8091 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.121.197 services:kv,index,n1ql,fts [_2] ip:172.23.121.198 services:kv,index,n1ql,fts [_3] ip:172.23.121.199 services:kv,index,n1ql,fts [_4] ip:172.23.121.200 services:kv,index,n1ql,fts [_5] ip:172.23.121.201 services:kv,index,n1ql,fts' [temp_rebalance_even_win2] $ /bin/sh -xe /tmp/jenkins2413467463083741764.sh + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + python3 testrunner.py -i /tmp/win10-bucket-ops-11111222223334444.ini -c conf/py-rebalance-extended-sanity.conf -p get-cbcollect-info=True,get-logs=True,get-coredumps=True INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: conf/py-rebalance-extended-sanity.conf INFO:__main__:Test prefix: rebalance.rebalancein.RebalanceInTests INFO:__main__:Test prefix: rebalance.rebalanceinout.RebalanceInOutTests INFO:__main__:Test prefix: rebalance.rebalanceout.RebalanceOutTests INFO:__main__:Test prefix: failover.failovertests.FailoverTests INFO:__main__:Test prefix: failover.AutoFailoverTests.AutoFailoverTests INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 18 INFO:__main__:--> Running test: rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=100000,GROUP=IN;P0 INFO:__main__:Logs folder: /data/workspace/temp_rebalance_even_win2/logs/testrunner-21-Apr-20_01-52-45/test_1 *** TestRunner *** {'cluster_name': 'win10-bucket-ops-11111222223334444', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'True', 'get-coredumps': 'True', 'get-logs': 'True', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'num_nodes': 5, 'spec': 'py-rebalance-extended-sanity'} Logs will be stored at /data/workspace/temp_rebalance_even_win2/logs/testrunner-21-Apr-20_01-52-45/test_1 ./testrunner -i /tmp/win10-bucket-ops-11111222223334444.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True -t rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=100000,GROUP=IN;P0 Test Input params: {'nodes_in': '1', 'nodes_init': '3', 'replicas': '1', 'items': '100000', 'GROUP': 'IN;P0', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'True', 'get-logs': 'True', 'get-coredumps': 'True', 'num_nodes': 5, 'case_number': 1, 'logs_folder': '/data/workspace/temp_rebalance_even_win2/logs/testrunner-21-Apr-20_01-52-45/test_1'} [2021-04-20 01:52:45,795] - [remote_util:299] INFO - SSH Connecting to 172.23.121.197 with username:Administrator, attempt#1 of 5 [2021-04-20 01:52:46,046] - [remote_util:335] INFO - SSH Connected to 172.23.121.197 as Administrator [2021-04-20 01:52:47,228] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-20 01:52:47,228] - [remote_util:3676] INFO - This is windows server! [2021-04-20 01:52:47,546] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-20 01:52:47,623] - [remote_util:1666] INFO - get windows information [2021-04-20 01:52:47,719] - [remote_util:3427] INFO - running command.raw on 172.23.121.197: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-20 01:52:47,978] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:48,288] - [remote_util:3427] INFO - running command.raw on 172.23.121.197: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-20 01:52:48,363] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:48,365] - [remote_util:938] INFO - 172.23.121.197 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-20 01:52:48,365] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-20 01:52:48,648] - [remote_util:169] INFO - process is running on 172.23.121.197: ['memcached.exe', '7488', 'Services', '0', '144,412', 'K'] [2021-04-20 01:52:48,649] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-20 01:52:49,034] - [remote_util:169] INFO - process is running on 172.23.121.197: ['erlsrv.exe', '2588', 'Services', '0', '3,140', 'K'] [2021-04-20 01:52:49,037] - [remote_util:299] INFO - SSH Connecting to 172.23.121.198 with username:Administrator, attempt#1 of 5 [2021-04-20 01:52:49,160] - [remote_util:335] INFO - SSH Connected to 172.23.121.198 as Administrator [2021-04-20 01:52:49,576] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-20 01:52:49,576] - [remote_util:3676] INFO - This is windows server! [2021-04-20 01:52:49,669] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-20 01:52:49,742] - [remote_util:1666] INFO - get windows information [2021-04-20 01:52:49,829] - [remote_util:3427] INFO - running command.raw on 172.23.121.198: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-20 01:52:50,004] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:50,082] - [remote_util:3427] INFO - running command.raw on 172.23.121.198: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-20 01:52:50,150] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:50,151] - [remote_util:938] INFO - 172.23.121.198 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-20 01:52:50,151] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-20 01:52:50,415] - [remote_util:169] INFO - process is running on 172.23.121.198: ['memcached.exe', '5032', 'Services', '0', '64,136', 'K'] [2021-04-20 01:52:50,416] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-20 01:52:50,700] - [remote_util:169] INFO - process is running on 172.23.121.198: ['erlsrv.exe', '2756', 'Services', '0', '3,140', 'K'] [2021-04-20 01:52:50,703] - [remote_util:299] INFO - SSH Connecting to 172.23.121.199 with username:Administrator, attempt#1 of 5 [2021-04-20 01:52:50,832] - [remote_util:335] INFO - SSH Connected to 172.23.121.199 as Administrator [2021-04-20 01:52:51,242] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-20 01:52:51,243] - [remote_util:3676] INFO - This is windows server! [2021-04-20 01:52:51,343] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-20 01:52:51,412] - [remote_util:1666] INFO - get windows information [2021-04-20 01:52:51,504] - [remote_util:3427] INFO - running command.raw on 172.23.121.199: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-20 01:52:51,687] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:51,760] - [remote_util:3427] INFO - running command.raw on 172.23.121.199: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-20 01:52:51,826] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:51,828] - [remote_util:938] INFO - 172.23.121.199 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-20 01:52:51,828] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-20 01:52:52,101] - [remote_util:169] INFO - process is running on 172.23.121.199: ['memcached.exe', '8004', 'Services', '0', '64,000', 'K'] [2021-04-20 01:52:52,102] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-20 01:52:52,390] - [remote_util:169] INFO - process is running on 172.23.121.199: ['erlsrv.exe', '7740', 'Services', '0', '3,132', 'K'] [2021-04-20 01:52:52,393] - [remote_util:299] INFO - SSH Connecting to 172.23.121.200 with username:Administrator, attempt#1 of 5 [2021-04-20 01:52:52,571] - [remote_util:335] INFO - SSH Connected to 172.23.121.200 as Administrator [2021-04-20 01:52:53,062] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-20 01:52:53,063] - [remote_util:3676] INFO - This is windows server! [2021-04-20 01:52:53,158] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-20 01:52:53,246] - [remote_util:1666] INFO - get windows information [2021-04-20 01:52:53,337] - [remote_util:3427] INFO - running command.raw on 172.23.121.200: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-20 01:52:53,594] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:53,735] - [remote_util:3427] INFO - running command.raw on 172.23.121.200: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-20 01:52:53,807] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:53,808] - [remote_util:938] INFO - 172.23.121.200 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-20 01:52:53,808] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-20 01:52:54,130] - [remote_util:169] INFO - process is running on 172.23.121.200: ['memcached.exe', '3668', 'Services', '0', '64,876', 'K'] [2021-04-20 01:52:54,131] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-20 01:52:54,500] - [remote_util:169] INFO - process is running on 172.23.121.200: ['erlsrv.exe', '2648', 'Services', '0', '3,124', 'K'] [2021-04-20 01:52:54,503] - [remote_util:299] INFO - SSH Connecting to 172.23.121.201 with username:Administrator, attempt#1 of 5 [2021-04-20 01:52:54,632] - [remote_util:335] INFO - SSH Connected to 172.23.121.201 as Administrator [2021-04-20 01:52:55,024] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-20 01:52:55,025] - [remote_util:3676] INFO - This is windows server! [2021-04-20 01:52:55,114] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-20 01:52:55,186] - [remote_util:1666] INFO - get windows information [2021-04-20 01:52:55,270] - [remote_util:3427] INFO - running command.raw on 172.23.121.201: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-20 01:52:55,455] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:55,531] - [remote_util:3427] INFO - running command.raw on 172.23.121.201: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-20 01:52:55,596] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-20 01:52:55,597] - [remote_util:938] INFO - 172.23.121.201 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-20 01:52:55,597] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-20 01:52:55,924] - [remote_util:169] INFO - process is running on 172.23.121.201: ['memcached.exe', '4604', 'Services', '0', '64,804', 'K'] [2021-04-20 01:52:55,924] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-20 01:52:56,211] - [remote_util:169] INFO - process is running on 172.23.121.201: ['erlsrv.exe', '4956', 'Services', '0', '3,164', 'K'] Run before suite setup for rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops lib/couchbase_helper/data_analysis_helper.py:824: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" pytests/security/SecretsMasterBase.py:63: DeprecationWarning: invalid escape sequence \. status, ipAddress = subprocess.getstatusoutput("ifconfig eth0 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | awk '{print $2}'") suite_setUp (rebalance.rebalancein.RebalanceInTests) ... -->before_suite_name:rebalance.rebalancein.RebalanceInTests.suite_setUp,suite: ]> 2021-04-20 01:52:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.197 with username:Administrator, attempt#1 of 5 2021-04-20 01:52:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.197 as Administrator 2021-04-20 01:52:57 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:52:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:52:57 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:52:57 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:52:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.197: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-10539-enterprise 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-10539-enterprise'] 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.197, nothing to delete 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.198, nothing to delete 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.199, nothing to delete 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.200, nothing to delete 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.201, nothing to delete 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.197:8091 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.197:8091 is running 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.198:8091 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.198:8091 is running 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.199:8091 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.199:8091 is running 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.200:8091 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.200:8091 is running 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.201:8091 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.201:8091 is running 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.121.197:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 100, 'mem_free': 3820929024, 'mem_total': 6437847040, 'swap_mem_used': 3103940608, 'swap_mem_total': 7511588864} 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.197 with username:Administrator, attempt#1 of 5 2021-04-20 01:52:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.197 as Administrator 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.197: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.198 with username:Administrator, attempt#1 of 5 2021-04-20 01:52:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.198 as Administrator 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.198: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.199 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.199 as Administrator 2021-04-20 01:53:01 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:01 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:01 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.199: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.200 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.200 as Administrator 2021-04-20 01:53:02 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:02 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.200: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.201 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.201 as Administrator 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:04 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:04 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.201: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.197, nothing to delete 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.198, nothing to delete 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.199, nothing to delete 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.200, nothing to delete 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.121.201, nothing to delete 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.197:8091 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.197:8091 is running 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.198:8091 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.198:8091 is running 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.199:8091 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.199:8091 is running 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.200:8091 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.200:8091 is running 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.121.201:8091 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.121.201:8091 is running 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-04-20 01:53:19 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.121.197 port:8091 ssh_username:Administrator, nodes/self 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '3097', 'memoryTotal': 6437847040, 'memoryFree': 3820273664, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.121.197:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10539-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2522, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.121.197', 'ip': '172.23.121.197', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.121.197 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.121.197 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2522 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.121.197,8091,['kv', 'index', 'n1ql', 'fts']) 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.121.197: 8091:hostname=172.23.121.197&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2021-04-20 01:53:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.121.197:8091//node/controller/setupServices body: hostname=172.23.121.197&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.121.197:8091:port=8091&username=Administrator&password=password 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.197 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.197 as Administrator 2021-04-20 01:53:21 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:21 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:21 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.197: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.197: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.197:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.197:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.121.198 port:8091 ssh_username:Administrator, nodes/self 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '2883', 'memoryTotal': 6437847040, 'memoryFree': 4243701760, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.121.198:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10539-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3290, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.121.198', 'ip': '172.23.121.198', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3290 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.121.198:8091:port=8091&username=Administrator&password=password 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.198 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.198 as Administrator 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.198: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.198: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.198:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.198:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.121.199 port:8091 ssh_username:Administrator, nodes/self 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '2568', 'memoryTotal': 6437847040, 'memoryFree': 4498034688, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.121.199:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10539-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3290, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.121.199', 'ip': '172.23.121.199', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3290 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.121.199:8091:port=8091&username=Administrator&password=password 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.199 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.199 as Administrator 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.199: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.199: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.199:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.199:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.121.200 port:8091 ssh_username:Administrator, nodes/self 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '2307', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 6139, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '172.23.121.200:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10539-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4113, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.121.200', 'ip': '172.23.121.200', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=4113 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-20 01:53:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.121.200:8091:port=8091&username=Administrator&password=password 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.200 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.200 as Administrator 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:25 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.200: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.200: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.200:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.200:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.121.201 port:8091 ssh_username:Administrator, nodes/self 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '423', 'memoryTotal': 6437847040, 'memoryFree': 4112785408, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.121.201:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10539-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3290, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@127.0.0.1', 'ip': '172.23.121.201', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3290 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.121.201:8091:port=8091&username=Administrator&password=password 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.201 with username:Administrator, attempt#1 of 5 2021-04-20 01:53:26 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.201 as Administrator 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.201: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.121.201: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.201:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.121.201:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-20 01:53:27 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-20 01:53:27 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.121.197 **** 2021-04-20 01:53:27 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-04-20 01:53:32 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-04-20 01:53:32 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-04-20 01:53:33 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.121.198:8091 to cluster 2021-04-20 01:53:33 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.121.198:8091 to this cluster @172.23.121.197:8091 2021-04-20 01:53:43 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2021-04-20 01:53:43 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-04-20 01:53:59 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.121.199:8091 to cluster 2021-04-20 01:53:59 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.121.199:8091 to this cluster @172.23.121.197:8091 2021-04-20 01:54:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2021-04-20 01:54:09 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-04-20 01:54:25 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.121.197,ns_1@172.23.121.198,ns_1@172.23.121.199', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-04-20 01:54:25 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-04-20 01:54:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-04-20 01:54:25 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-04-20 01:54:35 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.121.197: 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.197', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1618908867337, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {worker_died,\n {\'EXIT\',<0.15937.5>,\n {rebalance_failed,\n {service_error,\n <<"Unable to read index layout from cluster 127.0.0.1:8091. err = Invalid couchbase-server version">>}}}}}.\nRebalance Operation Id = 861e5e8febd107104b06aa918265a257', 'serverTime': '2021-04-20T01:54:27.337Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.199', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1618908865151, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-04-20T01:54:25.151Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.197', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1618908865012, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.121.197','ns_1@172.23.121.198',\n 'ns_1@172.23.121.199'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 861e5e8febd107104b06aa918265a257", 'serverTime': '2021-04-20T01:54:25.012Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.199', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1618908864970, 'shortText': 'message', 'text': 'Node ns_1@172.23.121.199 joined cluster', 'serverTime': '2021-04-20T01:54:24.970Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.199', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1618908864910, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.121.199\'. Version: "7.0.0-10539-enterprise".', 'serverTime': '2021-04-20T01:54:24.910Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.198', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1618908861960, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.121.198' saw that node 'ns_1@172.23.121.199' came up. Tags: []", 'serverTime': '2021-04-20T01:54:21.960Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.197', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1618908860522, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.121.197' saw that node 'ns_1@172.23.121.199' came up. Tags: []", 'serverTime': '2021-04-20T01:54:20.522Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.197', 'type': 'info', 'code': 0, 'module': 'ns_cluster', 'tstamp': 1618908859851, 'shortText': 'message', 'text': "Started node add transaction by adding node 'ns_1@172.23.121.199' to nodes_wanted (group: undefined)", 'serverTime': '2021-04-20T01:54:19.851Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.198', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1618908840800, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-04-20T01:54:00.800Z'} 2021-04-20 01:54:35 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.198', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1618908840619, 'shortText': 'message', 'text': 'Node ns_1@172.23.121.198 joined cluster', 'serverTime': '2021-04-20T01:54:00.619Z'} [, , , , , ] Tue Apr 20 01:54:35 2021 [>, , , , , , , , , , , , , , ] Cluster instance shutdown with force -->result: grabbing core dumps files from 172.23.121.197 2021-04-20 01:54:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.197 with username:Administrator, attempt#1 of 5 [, , , ] Tue Apr 20 01:54:35 2021 2021-04-20 01:54:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.197 as Administrator 2021-04-20 01:54:35 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:54:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:54:35 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:54:35 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.197: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.197: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.121.197 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process memcached on windows 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process is running on 172.23.121.197: ['memcached.exe', '7488', 'Services', '0', '144,388', 'K'] 2021-04-20 01:54:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process erl on windows 2021-04-20 01:54:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process is running on 172.23.121.197: ['erlsrv.exe', '2588', 'Services', '0', '3,140', 'K'] grabbing core dumps files from 172.23.121.198 2021-04-20 01:54:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.121.198 with username:Administrator, attempt#1 of 5 2021-04-20 01:54:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.121.198 as Administrator 2021-04-20 01:54:38 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-20 01:54:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-20 01:54:38 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-20 01:54:38 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-20 01:54:38 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.121.198: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat Build was aborted Aborted by Balakumaran G Archiving artifacts *** TestRunner *** Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: ABORTED