Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s22501 (P0 jython_slave PY3P0) in workspace /data/workspace/temp_win_private_job_3 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbase/testrunner > /usr/bin/git init /data/workspace/temp_win_private_job_3 # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 2fe4cf936aba56c7cf2c2396d556eb8462f304a4 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 2fe4cf936aba56c7cf2c2396d556eb8462f304a4 > /usr/bin/git rev-list 2fe4cf936aba56c7cf2c2396d556eb8462f304a4 # timeout=10 [temp_win_private_job_3] $ /bin/sh -xe /tmp/jenkins3940209981722914327.sh + echo '[global] username:Administrator password:Membase123 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 [_1] ip:172.23.107.126 port:8091 services:kv,index,n1ql,fts [_2] ip:172.23.107.127 port:8091 services:kv,index,n1ql,fts [_3] ip:172.23.107.129 port:8091 services:kv,index,n1ql,fts [_4] ip:172.23.107.131 port:8091 services:kv,index,n1ql,fts' + git submodule init Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' + git submodule update --init --force --remote Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out '8ae805783999ba5069f66511449402eda9d73931' + '[' -f /etc/redhat-release ']' + yum install -y maven Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, : subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: centos.mirror.shastacoe.net * extras: repos.eggycrew.com * updates: repos-lax.psychz.net Package maven-3.0.5-17.el7.noarch already installed and latest version Nothing to do + '[' -f /etc/lsb-release ']' + chmod 777 java_sdk_client/collections/configure.sh + mvn -f java_sdk_client/collections/pom.xml clean install OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building javaclient 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ javaclient --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ javaclient --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:2.3.1:compile (default-compile) @ javaclient --- [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 19 source files to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ javaclient --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /data/workspace/temp_win_private_job_3/java_sdk_client/collections/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.1:testCompile (default-testCompile) @ javaclient --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.10:test (default-test) @ javaclient --- [INFO] No tests to run. [INFO] Surefire report directory: /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-dependency-plugin:2.1:copy-dependencies (copy-dependencies) @ javaclient --- [INFO] Copying core-io-2.1.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/core-io-2.1.3.jar [INFO] Copying java-client-3.1.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/java-client-3.1.3.jar [INFO] Copying jackson-annotations-2.10.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/jackson-annotations-2.10.3.jar [INFO] Copying jackson-core-2.10.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/jackson-core-2.10.3.jar [INFO] Copying jackson-databind-2.10.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/jackson-databind-2.10.3.jar [INFO] Copying javafaker-1.0.2.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/javafaker-1.0.2.jar [INFO] Copying generex-1.0.2.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/generex-1.0.2.jar [INFO] Copying json-simple-1.1.1.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/json-simple-1.1.1.jar [INFO] Copying commons-codec-1.11.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/commons-codec-1.11.jar [INFO] Copying commons-logging-1.2.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/commons-logging-1.2.jar [INFO] Copying automaton-1.11-8.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/automaton-1.11-8.jar [INFO] Copying reactor-core-3.4.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/reactor-core-3.4.3.jar [INFO] Copying rxjava-2.2.0.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/rxjava-2.2.0.jar [INFO] Copying junit-4.12.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/junit-4.12.jar [INFO] Copying log4j-1.2.17.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/log4j-1.2.17.jar [INFO] Copying argparse4j-0.8.1.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/argparse4j-0.8.1.jar [INFO] Copying commons-lang3-3.5.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/commons-lang3-3.5.jar [INFO] Copying httpclient-4.5.12.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/httpclient-4.5.12.jar [INFO] Copying httpcore-4.4.13.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/httpcore-4.4.13.jar [INFO] Copying hamcrest-core-1.3.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/hamcrest-core-1.3.jar [INFO] Copying reactive-streams-1.0.2.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/reactive-streams-1.0.2.jar [INFO] Copying slf4j-api-1.7.30.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/slf4j-api-1.7.30.jar [INFO] Copying slf4j-log4j12-1.7.30.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/slf4j-log4j12-1.7.30.jar [INFO] Copying snakeyaml-1.23-android.jar to /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/lib/snakeyaml-1.23-android.jar [INFO] [INFO] --- maven-resources-plugin:2.6:copy-resources (copy-resources) @ javaclient --- [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /data/workspace/temp_win_private_job_3/java_sdk_client/collections/resources [INFO] [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ javaclient --- [INFO] Building jar: /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/javaclient.jar [INFO] [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ javaclient --- [INFO] Installing /data/workspace/temp_win_private_job_3/java_sdk_client/collections/target/javaclient/javaclient.jar to /root/.m2/repository/com/couchbase/javaclient/javaclient/1.0-SNAPSHOT/javaclient-1.0-SNAPSHOT.jar [INFO] Installing /data/workspace/temp_win_private_job_3/java_sdk_client/collections/pom.xml to /root/.m2/repository/com/couchbase/javaclient/javaclient/1.0-SNAPSHOT/javaclient-1.0-SNAPSHOT.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.196s [INFO] Finished at: Fri Apr 30 04:40:24 PDT 2021 [INFO] Final Memory: 23M/370M [INFO] ------------------------------------------------------------------------ [temp_win_private_job_3] $ /bin/sh -xe /tmp/jenkins2445696035859353629.sh + python3 testrunner.py -i /tmp/testexec-temp_win_private_job_3.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True,skip_cleanup=True -t rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=100000,skip_cleanup=True INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=100000,skip_cleanup=True INFO:__main__:Logs folder: /data/workspace/temp_win_private_job_3/logs/testrunner-21-Apr-30_04-40-25/test_1 *** TestRunner *** {'cluster_name': 'testexec-temp_win_private_job_3', 'get-cbcollect-info': 'True', 'get-coredumps': 'True', 'get-logs': 'True', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'num_nodes': 4, 'skip_cleanup': 'True'} Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-Apr-30_04-40-25/test_1 ./testrunner -i /tmp/testexec-temp_win_private_job_3.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True,skip_cleanup=True -t rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=100000,skip_cleanup=True Test Input params: {'nodes_in': '1', 'nodes_init': '3', 'replicas': '1', 'items': '100000', 'skip_cleanup': 'True', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'cluster_name': 'testexec-temp_win_private_job_3', 'get-cbcollect-info': 'True', 'get-logs': 'True', 'get-coredumps': 'True', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-Apr-30_04-40-25/test_1'} [2021-04-30 04:40:25,810] - [remote_util:299] INFO - SSH Connecting to 172.23.107.126 with username:Administrator, attempt#1 of 5 [2021-04-30 04:40:26,015] - [remote_util:335] INFO - SSH Connected to 172.23.107.126 as Administrator [2021-04-30 04:40:26,602] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-30 04:40:26,603] - [remote_util:3676] INFO - This is windows server! [2021-04-30 04:40:26,695] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-30 04:40:26,773] - [remote_util:1666] INFO - get windows information [2021-04-30 04:40:26,880] - [remote_util:3427] INFO - running command.raw on 172.23.107.126: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-30 04:40:27,097] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:27,179] - [remote_util:3427] INFO - running command.raw on 172.23.107.126: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-30 04:40:27,291] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:27,292] - [remote_util:938] INFO - 172.23.107.126 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-30 04:40:27,293] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-30 04:40:27,609] - [remote_util:169] INFO - process is running on 172.23.107.126: ['memcached.exe', '6140', 'Services', '0', '55,556', 'K'] [2021-04-30 04:40:27,609] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-30 04:40:27,872] - [remote_util:169] INFO - process is running on 172.23.107.126: ['erlsrv.exe', '828', 'Services', '0', '3,080', 'K'] [2021-04-30 04:40:27,874] - [remote_util:299] INFO - SSH Connecting to 172.23.107.127 with username:Administrator, attempt#1 of 5 [2021-04-30 04:40:28,008] - [remote_util:335] INFO - SSH Connected to 172.23.107.127 as Administrator [2021-04-30 04:40:28,389] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-30 04:40:28,389] - [remote_util:3676] INFO - This is windows server! [2021-04-30 04:40:28,473] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-30 04:40:28,545] - [remote_util:1666] INFO - get windows information [2021-04-30 04:40:28,626] - [remote_util:3427] INFO - running command.raw on 172.23.107.127: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-30 04:40:28,794] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:28,874] - [remote_util:3427] INFO - running command.raw on 172.23.107.127: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-30 04:40:28,943] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:28,944] - [remote_util:938] INFO - 172.23.107.127 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-30 04:40:28,945] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-30 04:40:29,188] - [remote_util:169] INFO - process is running on 172.23.107.127: ['memcached.exe', '5660', 'Services', '0', '56,280', 'K'] [2021-04-30 04:40:29,188] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-30 04:40:29,462] - [remote_util:169] INFO - process is running on 172.23.107.127: ['erlsrv.exe', '4172', 'Services', '0', '3,164', 'K'] [2021-04-30 04:40:29,465] - [remote_util:299] INFO - SSH Connecting to 172.23.107.129 with username:Administrator, attempt#1 of 5 [2021-04-30 04:40:29,588] - [remote_util:335] INFO - SSH Connected to 172.23.107.129 as Administrator [2021-04-30 04:40:29,988] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-30 04:40:29,988] - [remote_util:3676] INFO - This is windows server! [2021-04-30 04:40:30,078] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-30 04:40:30,151] - [remote_util:1666] INFO - get windows information [2021-04-30 04:40:30,227] - [remote_util:3427] INFO - running command.raw on 172.23.107.129: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-30 04:40:30,566] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:30,688] - [remote_util:3427] INFO - running command.raw on 172.23.107.129: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-30 04:40:30,754] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:30,755] - [remote_util:938] INFO - 172.23.107.129 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-30 04:40:30,755] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-30 04:40:30,998] - [remote_util:169] INFO - process is running on 172.23.107.129: ['memcached.exe', '5280', 'Services', '0', '54,800', 'K'] [2021-04-30 04:40:30,998] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-30 04:40:31,263] - [remote_util:169] INFO - process is running on 172.23.107.129: ['erlsrv.exe', '5964', 'Services', '0', '3,168', 'K'] [2021-04-30 04:40:31,265] - [remote_util:299] INFO - SSH Connecting to 172.23.107.131 with username:Administrator, attempt#1 of 5 [2021-04-30 04:40:31,387] - [remote_util:335] INFO - SSH Connected to 172.23.107.131 as Administrator [2021-04-30 04:40:31,773] - [remote_util:1626] INFO - File /cygdrive/c/Windows/win.ini was found [2021-04-30 04:40:31,773] - [remote_util:3676] INFO - This is windows server! [2021-04-30 04:40:31,855] - [remote_util:1626] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2021-04-30 04:40:31,931] - [remote_util:1666] INFO - get windows information [2021-04-30 04:40:32,011] - [remote_util:3427] INFO - running command.raw on 172.23.107.131: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat [2021-04-30 04:40:32,343] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:32,421] - [remote_util:3427] INFO - running command.raw on 172.23.107.131: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' [2021-04-30 04:40:32,487] - [remote_util:3475] INFO - command executed successfully with Administrator [2021-04-30 04:40:32,488] - [remote_util:938] INFO - 172.23.107.131 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists [2021-04-30 04:40:32,488] - [remote_util:157] INFO - Checking for process memcached on windows [2021-04-30 04:40:32,714] - [remote_util:169] INFO - process is running on 172.23.107.131: ['memcached.exe', '4356', 'Services', '0', '54,768', 'K'] [2021-04-30 04:40:32,715] - [remote_util:157] INFO - Checking for process erl on windows [2021-04-30 04:40:32,962] - [remote_util:169] INFO - process is running on 172.23.107.131: ['erlsrv.exe', '6116', 'Services', '0', '3,144', 'K'] Run before suite setup for rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops lib/couchbase_helper/data_analysis_helper.py:826: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" pytests/security/SecretsMasterBase.py:63: DeprecationWarning: invalid escape sequence \. status, ipAddress = subprocess.getstatusoutput("ifconfig eth0 | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | awk '{print $2}'") suite_setUp (rebalance.rebalancein.RebalanceInTests) ... -->before_suite_name:rebalance.rebalancein.RebalanceInTests.suite_setUp,suite: ]> 2021-04-30 04:40:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.126 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.126 as Administrator 2021-04-30 04:40:33 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:33 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.126: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-10544-enterprise 2021-04-30 04:40:34 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.107.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-04-30 04:40:34 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.107.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.126 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.126 as Administrator 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:34 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.126: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.127 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.127 as Administrator 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:35 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.127: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.129 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.129 as Administrator 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:36 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.129: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.131 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.131 as Administrator 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.131: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:38 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:47 | WARNING | MainProcess | MainThread | [basetestcase.tearDown] CLEANUP WAS SKIPPED 2021-04-30 04:40:47 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-04-30 04:40:47 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2021-04-30 04:40:47 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-04-30 04:40:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.126 port:8091 ssh_username:Administrator, nodes/self 2021-04-30 04:40:48 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '82', 'memoryTotal': 6437847040, 'memoryFree': 4202950656, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.107.126:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10544-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3017, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.126', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-30 04:40:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-30 04:40:48 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2021-04-30 04:40:48 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.107.126 2021-04-30 04:40:48 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.107.126 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2522 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.107.126,8091,['kv', 'index', 'n1ql', 'fts']) 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.107.126: 8091:hostname=172.23.107.126&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.126:8091:port=8091&username=Administrator&password=password 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.126 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.126 as Administrator 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.126: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.126:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.126:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.127 port:8091 ssh_username:Administrator, nodes/self 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '752', 'memoryTotal': 6437847040, 'memoryFree': 4754345984, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.107.127:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10544-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3017, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.127', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-30 04:40:51 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3290 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.127:8091:port=8091&username=Administrator&password=password 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.127 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.127 as Administrator 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.127: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.127:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.127:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.129 port:8091 ssh_username:Administrator, nodes/self 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '749', 'memoryTotal': 6437847040, 'memoryFree': 4453552128, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.107.129:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10544-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3017, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.129', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-30 04:40:52 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.129:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3290 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.129:8091:port=8091&username=Administrator&password=password 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-30 04:40:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.129 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.129 as Administrator 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.129: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.129: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.129:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.129:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.131 port:8091 ssh_username:Administrator, nodes/self 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '753', 'memoryTotal': 6437847040, 'memoryFree': 4421881856, 'mcdMemoryReserved': 4911, 'mcdMemoryAllocated': 4911, 'status': 'healthy', 'hostname': '172.23.107.131:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-10544-enterprise', 'os': 'win64', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3017, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.131', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 6139, 'curr_items': 0} 2021-04-30 04:40:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.131:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=3290 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.131:8091:port=8091&username=Administrator&password=password 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.131 with username:Administrator, attempt#1 of 5 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.131 as Administrator 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:40:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.find_windows_info] get windows information 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.131: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.131: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.131:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.131:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-04-30 04:40:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-04-30 04:40:55 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.107.126 **** 2021-04-30 04:40:55 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.107.126:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-04-30 04:40:55 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-04-30 04:40:56 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-04-30 04:41:01 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-04-30 04:41:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-04-30 04:41:01 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.127:8091 to cluster 2021-04-30 04:41:01 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.127:8091 to this cluster @172.23.107.126:8091 2021-04-30 04:41:11 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.03 seconds 2021-04-30 04:41:11 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-04-30 04:41:26 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.129:8091 to cluster 2021-04-30 04:41:26 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.129:8091 to this cluster @172.23.107.126:8091 2021-04-30 04:41:36 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2021-04-30 04:41:36 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-04-30 04:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.107.126,ns_1@172.23.107.127,ns_1@172.23.107.129', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-04-30 04:41:56 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2021-04-30 04:42:06 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.107.126: 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.126', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1619782918792, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {worker_died,\n {\'EXIT\',<0.7815.0>,\n {rebalance_failed,\n {service_error,\n <<"Unable to read index layout from cluster 127.0.0.1:8091. err = Invalid couchbase-server version">>}}}}}.\nRebalance Operation Id = 0455cdfb859655785273a70abbb9e2e2', 'serverTime': '2021-04-30T04:41:58.792Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.129', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1619782916904, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-04-30T04:41:56.904Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.126', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1619782916783, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.107.126','ns_1@172.23.107.127',\n 'ns_1@172.23.107.129'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 0455cdfb859655785273a70abbb9e2e2", 'serverTime': '2021-04-30T04:41:56.783Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.129', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1619782916682, 'shortText': 'message', 'text': 'Node ns_1@172.23.107.129 joined cluster', 'serverTime': '2021-04-30T04:41:56.682Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.127', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1619782916623, 'shortText': 'message', 'text': 'Conflicting configuration changes to field scramsha_fallback_salt:\n<<61,25,203,206,189,214,210,168,156,185,158,187>> and\n<<67,8,217,33,25,235,34,11,109,107,105,126>>, choosing the former.', 'serverTime': '2021-04-30T04:41:56.623Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.129', 'type': 'info', 'code': 1, 'module': 'menelaus_web_sup', 'tstamp': 1619782916620, 'shortText': 'web start ok', 'text': 'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.107.129\'. Version: "7.0.0-10544-enterprise".', 'serverTime': '2021-04-30T04:41:56.620Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.126', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1619782916615, 'shortText': 'message', 'text': 'Conflicting configuration changes to field scramsha_fallback_salt:\n<<61,25,203,206,189,214,210,168,156,185,158,187>> and\n<<67,8,217,33,25,235,34,11,109,107,105,126>>, choosing the former.', 'serverTime': '2021-04-30T04:41:56.615Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.129', 'type': 'info', 'code': 0, 'module': 'ns_config', 'tstamp': 1619782916584, 'shortText': 'message', 'text': 'Conflicting configuration changes to field scramsha_fallback_salt:\n<<61,25,203,206,189,214,210,168,156,185,158,187>> and\n<<67,8,217,33,25,235,34,11,109,107,105,126>>, choosing the former.', 'serverTime': '2021-04-30T04:41:56.584Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.127', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1619782908179, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.107.127' saw that node 'ns_1@172.23.107.129' came up. Tags: []", 'serverTime': '2021-04-30T04:41:48.179Z'} 2021-04-30 04:42:06 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {'node': 'ns_1@172.23.107.126', 'type': 'info', 'code': 4, 'module': 'ns_node_disco', 'tstamp': 1619782908019, 'shortText': 'node up', 'text': "Node 'ns_1@172.23.107.126' saw that node 'ns_1@172.23.107.129' came up. Tags: []", 'serverTime': '2021-04-30T04:41:48.019Z'} [, , , , , ] Fri Apr 30 04:42:06 2021 [>, , , , , , , , , , , , , , ] Cluster instance shutdown with force -->result: grabbing core dumps files from 172.23.107.126 2021-04-30 04:42:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.126 with username:Administrator, attempt#1 of 5 [, , , ] Fri Apr 30 04:42:06 2021 2021-04-30 04:42:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.126 as Administrator 2021-04-30 04:42:07 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:42:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:42:07 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:42:07 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:42:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.126: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:42:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:42:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.126: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' 2021-04-30 04:42:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:42:08 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.126 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists 2021-04-30 04:42:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process memcached on windows 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process is running on 172.23.107.126: ['memcached.exe', '6140', 'Services', '0', '55,948', 'K'] 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process erl on windows 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process is running on 172.23.107.126: ['erlsrv.exe', '828', 'Services', '0', '2,956', 'K'] grabbing core dumps files from 172.23.107.127 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.127 with username:Administrator, attempt#1 of 5 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.127 as Administrator 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:42:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.127: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.127: cat '/cygdrive/c/Program Files/Couchbase/Server/VERSION.txt' 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.107.127 **** The version file /cygdrive/c/Program\ Files/Couchbase/Server/ VERSION.txt exists 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process memcached on windows 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process is running on 172.23.107.127: ['memcached.exe', '5660', 'Services', '0', '56,616', 'K'] 2021-04-30 04:42:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process erl on windows 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process is running on 172.23.107.127: ['erlsrv.exe', '4172', 'Services', '0', '3,164', 'K'] grabbing core dumps files from 172.23.107.129 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.107.129 with username:Administrator, attempt#1 of 5 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.107.129 as Administrator 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/Windows/win.ini was found 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] This is windows server! 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.find_file] File /cygdrive/c/tmp/windows_info.txt was found 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.find_windows_info] get windows information 2021-04-30 04:42:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.107.129: echo "ipconfig" > /tmp/cmd.bat; chmod u=rwx /tmp/cmd.bat; /tmp/cmd.bat Build was aborted Aborted by Balakumaran G Archiving artifacts *** TestRunner *** Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: ABORTED