Started by upstream project "build_sanity_matrix" build number 9106 originally caused by: Started by upstream project "build-sanity-trigger" build number 9960 originally caused by: Started by timer with parameters: {VERSION=7.0.0} Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on build-sanity-tester-3 (sanity-testers) in workspace /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... [WS-CLEANUP] Done The recommended git tool is: NONE No credentials specified Cloning the remote Git repository Cloning repository git://github.com/couchbase/testrunner.git > git init /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1 # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner.git > git --version # timeout=10 > git --version # 'git version 2.24.1' > git fetch --tags --force --progress -- git://github.com/couchbase/testrunner.git +refs/heads/*:refs/remotes/origin/* # timeout=20 > git config remote.origin.url git://github.com/couchbase/testrunner.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision ccd490060146e261574739082e75b7573c497824 (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f ccd490060146e261574739082e75b7573c497824 # timeout=10 Commit message: "do not form cluster in bkrs bwc test. Add new 6 nodes template brks-bwc" > git rev-list --no-walk ccd490060146e261574739082e75b7573c497824 # timeout=10 No emails were triggered. [EnvInject] - Executing scripts and injecting environment variables after the SCM step. New run name is '#9106--7.0.0-5148' [4node_1] $ /bin/bash -x /tmp/jenkins6239551877371304978.sh + echo version=7.0.0 version=7.0.0 + cd /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1 + '[' master = master ']' + echo 'Init and update java sdk client submodule for collections...' Init and update java sdk client submodule for collections... + git submodule init Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' + git submodule update --init --force --remote Cloning into '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/java_sdk_client'... Submodule path 'java_sdk_client': checked out '8ae805783999ba5069f66511449402eda9d73931' + d=centos7 + '[' centos7 == ubuntu12 ']' + '[' centos7 == windows ']' + c=0 + '[' 4node_1 = 1node ']' + '[' 4node_1 = 4node ']' + '[' 4node_1 = 4node_1 ']' + c=4 + cd conf + split -l 31 py-multi-node-sanity.conf + mv xaa py-multi-node-sanity-1.conf + mv xab py-multi-node-sanity-2.conf + cd /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1 + v=7.0.0-5148 + cat + '[' centos7 == win64 ']' ++ curl -d @data.json -X POST -H 'Content-Type: application/json' http://dbapi.build.couchbase.com:8000/v1/reservations % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 170 100 81 100 89 3131 3440 --:--:-- --:--:-- --:--:-- 3560 + IP_LIST='{"vms": ["172.23.105.151", "172.23.105.153", "172.23.107.160", "172.23.107.184"]}' ++ echo '{"vms":' '["172.23.105.151",' '"172.23.105.153",' '"172.23.107.160",' '"172.23.107.184"]}' ++ python -c 'import json, sys; j=json.load(sys.stdin); print '\'' '\''.join(j['\''vms'\''])' + IPS='172.23.105.151 172.23.105.153 172.23.107.160 172.23.107.184' + echo 'IPS=172.23.105.151 172.23.105.153 172.23.107.160 172.23.107.184' IPS=172.23.105.151 172.23.105.153 172.23.107.160 172.23.107.184 + counter=10 + '[' -z '172.23.105.151 172.23.105.153 172.23.107.160 172.23.107.184' ']' + rm -f data.json + rm -rf build + git clone git://github.com/couchbase/build.git Cloning into 'build'... + EXTRA_INSTALL_PARAMS=fts_query_limit=10000000 + NODE_IPS='172.23.105.151 172.23.105.153 172.23.107.160 172.23.107.184' + build/scripts/jenkins/system_test/build-sanity/run_sanity.sh NODE CONFIGURATION: [global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [_1] ip:172.23.105.151 services:kv,index,n1ql,fts,backup [cluster1] 1:_1 2:_2 [cluster2] 1:_3 2:_4 [servers] 1:_1 2:_2 3:_3 4:_4 [_2] ip:172.23.105.153 port:8091 [_3] ip:172.23.107.160 services:kv,index,n1ql,fts,backup port:8091 [_4] ip:172.23.107.184 port:8091 services:kv,cbas version=7.0.0-5148 7.0.0-5148 Running: COUCHBASE_NUM_VBUCKETS=64 python scripts/new_install.py -i node_conf.ini -p version=7.0.0-5148,product=cb,parallel=True,fts_query_limit=10000000 2021-05-11 17:00:12,527 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:00:12,657 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:00:13,115 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:13,510 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:13,512 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:00:13,657 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:00:14,122 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:14,526 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:14,529 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:00:14,637 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:00:15,066 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:15,504 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:15,507 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:00:15,756 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:00:16,325 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:16,839 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:16,842 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:00:17,080 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:00:17,630 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:18,155 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:18,157 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:00:18,397 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:00:18,901 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:19,368 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:19,371 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:00:19,537 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:00:19,896 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:20,308 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:20,310 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:00:20,511 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:00:21,014 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:00:21,431 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:00:21,432 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:21,439 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm is live 2021-05-11 17:00:21,439 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:21,440 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm is live 2021-05-11 17:00:21,440 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:21,451 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm is live 2021-05-11 17:00:21,451 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:21,462 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm is live 2021-05-11 17:00:21,463 - root - INFO - Downloading build binary to 172.23.105.151:/tmp/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm.. 2021-05-11 17:00:38,062 - root - INFO - running command.raw on 172.23.105.151: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-05-11 17:00:38,145 - root - INFO - command executed successfully with root 2021-05-11 17:00:38,145 - root - INFO - running command.raw on 172.23.105.151: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:38,214 - root - INFO - command executed successfully with root 2021-05-11 17:00:38,279 - root - INFO - Downloading build binary to 172.23.105.153:/tmp/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm.. 2021-05-11 17:00:42,995 - root - INFO - running command.raw on 172.23.105.153: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-05-11 17:00:43,039 - root - INFO - command executed successfully with root 2021-05-11 17:00:43,040 - root - INFO - running command.raw on 172.23.105.153: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:43,113 - root - INFO - command executed successfully with root 2021-05-11 17:00:43,187 - root - INFO - Downloading build binary to 172.23.107.160:/tmp/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm.. 2021-05-11 17:00:50,407 - root - INFO - running command.raw on 172.23.107.160: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-05-11 17:00:50,440 - root - INFO - command executed successfully with root 2021-05-11 17:00:50,441 - root - INFO - running command.raw on 172.23.107.160: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:50,507 - root - INFO - command executed successfully with root 2021-05-11 17:00:50,576 - root - INFO - Downloading build binary to 172.23.107.184:/tmp/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm.. 2021-05-11 17:00:56,598 - root - INFO - running command.raw on 172.23.107.184: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/5148/couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-05-11 17:00:56,634 - root - INFO - command executed successfully with root 2021-05-11 17:00:56,635 - root - INFO - running command.raw on 172.23.107.184: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-5148-centos7.x86_64.rpm 2021-05-11 17:00:56,704 - root - INFO - command executed successfully with root 2021-05-11 17:01:02,554 - root - INFO - Done with uninstall on 172.23.105.153. 2021-05-11 17:01:04,755 - root - INFO - Done with uninstall on 172.23.105.151. 2021-05-11 17:01:16,624 - root - INFO - Done with uninstall on 172.23.107.160. 2021-05-11 17:01:18,085 - root - INFO - Done with uninstall on 172.23.107.184. 2021-05-11 17:03:00,639 - root - INFO - Done with install on 172.23.105.153. 2021-05-11 17:03:22,257 - root - INFO - Done with install on 172.23.105.151. 2021-05-11 17:03:35,644 - root - INFO - Done with install on 172.23.107.184. 2021-05-11 17:03:37,778 - root - INFO - Done with install on 172.23.107.160. 2021-05-11 17:04:00,700 - root - INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.153 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-05-11 17:04:01,186 - root - INFO - command executed successfully with root 2021-05-11 17:04:01,260 - root - INFO - running command.raw on 172.23.105.153: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server; grep bleveMaxResultWindow=10000000 /opt/couchbase/bin/couchbase-server > /dev/null && echo 1 || echo 0 2021-05-11 17:04:01,301 - root - INFO - command executed successfully with root 2021-05-11 17:04:01,405 - root - INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:01,515 - root - INFO - Running systemd stop command on this server 2021-05-11 17:04:01,515 - root - INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service 2021-05-11 17:04:03,364 - root - INFO - command executed successfully with root 2021-05-11 17:04:03,413 - root - INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:03,559 - root - INFO - Running systemd start command on this server 2021-05-11 17:04:03,559 - root - INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service 2021-05-11 17:04:03,596 - root - INFO - command executed successfully with root 2021-05-11 17:04:13,607 - root - INFO - fts_query_limit set to 10000000 on 172.23.105.153 2021-05-11 17:04:13,613 - root - ERROR - socket error while connecting to http://172.23.105.153:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:16,622 - root - ERROR - socket error while connecting to http://172.23.105.153:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:22,309 - root - INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.151 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-05-11 17:04:22,762 - root - INFO - command executed successfully with root 2021-05-11 17:04:22,866 - root - INFO - running command.raw on 172.23.105.151: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server; grep bleveMaxResultWindow=10000000 /opt/couchbase/bin/couchbase-server > /dev/null && echo 1 || echo 0 2021-05-11 17:04:22,901 - root - INFO - command executed successfully with root 2021-05-11 17:04:22,998 - root - INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:23,168 - root - INFO - Running systemd stop command on this server 2021-05-11 17:04:23,168 - root - INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service 2021-05-11 17:04:23,658 - root - INFO - Setting KV memory quota as 4186 MB on 172.23.105.153 2021-05-11 17:04:23,658 - root - INFO - pools/default params : memoryQuota=4186 2021-05-11 17:04:23,668 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-05-11 17:04:23,669 - root - INFO - /node/controller/setupServices params on 172.23.105.153: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-05-11 17:04:23,700 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:04:23,700 - root - INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password 2021-05-11 17:04:23,818 - root - INFO - --> status:True 2021-05-11 17:04:23,819 - root - INFO - Done with init on 172.23.105.153. 2021-05-11 17:04:23,957 - root - INFO - Done with cleanup on 172.23.105.153. 2021-05-11 17:04:24,978 - root - INFO - command executed successfully with root 2021-05-11 17:04:25,024 - root - INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:25,167 - root - INFO - Running systemd start command on this server 2021-05-11 17:04:25,168 - root - INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service 2021-05-11 17:04:25,201 - root - INFO - command executed successfully with root 2021-05-11 17:04:35,211 - root - INFO - fts_query_limit set to 10000000 on 172.23.105.151 2021-05-11 17:04:35,215 - root - ERROR - socket error while connecting to http://172.23.105.151:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:35,691 - root - INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.107.184 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-05-11 17:04:36,142 - root - INFO - command executed successfully with root 2021-05-11 17:04:36,213 - root - INFO - running command.raw on 172.23.107.184: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server; grep bleveMaxResultWindow=10000000 /opt/couchbase/bin/couchbase-server > /dev/null && echo 1 || echo 0 2021-05-11 17:04:36,236 - root - INFO - command executed successfully with root 2021-05-11 17:04:36,302 - root - INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:36,375 - root - INFO - Running systemd stop command on this server 2021-05-11 17:04:36,375 - root - INFO - running command.raw on 172.23.107.184: systemctl stop couchbase-server.service 2021-05-11 17:04:37,827 - root - INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.107.160 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-05-11 17:04:38,220 - root - ERROR - socket error while connecting to http://172.23.105.151:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:38,227 - root - INFO - command executed successfully with root 2021-05-11 17:04:38,296 - root - INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:38,308 - root - INFO - command executed successfully with root 2021-05-11 17:04:38,373 - root - INFO - Running systemd start command on this server 2021-05-11 17:04:38,373 - root - INFO - running command.raw on 172.23.107.184: systemctl start couchbase-server.service 2021-05-11 17:04:38,383 - root - INFO - running command.raw on 172.23.107.160: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server; grep bleveMaxResultWindow=10000000 /opt/couchbase/bin/couchbase-server > /dev/null && echo 1 || echo 0 2021-05-11 17:04:38,402 - root - INFO - command executed successfully with root 2021-05-11 17:04:38,404 - root - INFO - command executed successfully with root 2021-05-11 17:04:38,468 - root - INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:38,532 - root - INFO - Running systemd stop command on this server 2021-05-11 17:04:38,533 - root - INFO - running command.raw on 172.23.107.160: systemctl stop couchbase-server.service 2021-05-11 17:04:40,380 - root - INFO - command executed successfully with root 2021-05-11 17:04:40,403 - root - INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-05-11 17:04:40,463 - root - INFO - Running systemd start command on this server 2021-05-11 17:04:40,463 - root - INFO - running command.raw on 172.23.107.160: systemctl start couchbase-server.service 2021-05-11 17:04:40,486 - root - INFO - command executed successfully with root 2021-05-11 17:04:45,247 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.105.151 2021-05-11 17:04:45,247 - root - INFO - pools/default params : indexMemoryQuota=256 2021-05-11 17:04:45,259 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.105.151 2021-05-11 17:04:45,259 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-05-11 17:04:45,266 - root - INFO - Setting KV memory quota as 3419 MB on 172.23.105.151 2021-05-11 17:04:45,267 - root - INFO - pools/default params : memoryQuota=3419 2021-05-11 17:04:45,272 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) 2021-05-11 17:04:45,272 - root - INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup 2021-05-11 17:04:45,304 - root - INFO - settings/indexes params : storageMode=plasma 2021-05-11 17:04:45,313 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:04:45,313 - root - INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password 2021-05-11 17:04:45,423 - root - INFO - --> status:True 2021-05-11 17:04:45,424 - root - INFO - Done with init on 172.23.105.151. 2021-05-11 17:04:45,569 - root - INFO - Done with cleanup on 172.23.105.151. 2021-05-11 17:04:48,412 - root - INFO - fts_query_limit set to 10000000 on 172.23.107.184 2021-05-11 17:04:48,415 - root - ERROR - socket error while connecting to http://172.23.107.184:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:50,497 - root - INFO - fts_query_limit set to 10000000 on 172.23.107.160 2021-05-11 17:04:50,499 - root - ERROR - socket error while connecting to http://172.23.107.160:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:51,419 - root - ERROR - socket error while connecting to http://172.23.107.184:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:53,503 - root - ERROR - socket error while connecting to http://172.23.107.160:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 17:04:58,439 - root - INFO - Setting CBAS memory quota as 1024 MB on 172.23.107.184 2021-05-11 17:04:58,439 - root - INFO - pools/default params : cbasMemoryQuota=1024 2021-05-11 17:04:58,450 - root - INFO - Setting KV memory quota as 3163 MB on 172.23.107.184 2021-05-11 17:04:58,450 - root - INFO - pools/default params : memoryQuota=3163 2021-05-11 17:04:58,455 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'cbas']) 2021-05-11 17:04:58,455 - root - INFO - /node/controller/setupServices params on 172.23.107.184: 8091:hostname=None&user=Administrator&password=password&services=kv%2Ccbas 2021-05-11 17:04:58,489 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:04:58,489 - root - INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password 2021-05-11 17:04:58,592 - root - INFO - --> status:True 2021-05-11 17:04:58,592 - root - INFO - Done with init on 172.23.107.184. 2021-05-11 17:04:58,734 - root - INFO - Done with cleanup on 172.23.107.184. 2021-05-11 17:05:00,526 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.107.160 2021-05-11 17:05:00,526 - root - INFO - pools/default params : indexMemoryQuota=256 2021-05-11 17:05:00,536 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.107.160 2021-05-11 17:05:00,536 - root - INFO - pools/default params : ftsMemoryQuota=512 2021-05-11 17:05:00,540 - root - INFO - Setting KV memory quota as 3287 MB on 172.23.107.160 2021-05-11 17:05:00,540 - root - INFO - pools/default params : memoryQuota=3287 2021-05-11 17:05:00,546 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) 2021-05-11 17:05:00,546 - root - INFO - /node/controller/setupServices params on 172.23.107.160: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup 2021-05-11 17:05:00,580 - root - INFO - settings/indexes params : storageMode=plasma 2021-05-11 17:05:00,588 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:05:00,588 - root - INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password 2021-05-11 17:05:00,689 - root - INFO - --> status:True 2021-05-11 17:05:00,690 - root - INFO - Done with init on 172.23.107.160. 2021-05-11 17:05:00,804 - root - INFO - Done with cleanup on 172.23.107.160. 2021-05-11 17:05:16,984 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 17:05:17,013 - root - INFO - cluster:C1 node:172.23.105.151:8091 version:7.0.0-5148-enterprise aFamily:inet services:['backup', 'fts', 'index', 'kv', 'n1ql'] 2021-05-11 17:05:17,013 - root - INFO - cluster:C2 node:172.23.105.153:8091 version:7.0.0-5148-enterprise aFamily:inet services:['kv'] 2021-05-11 17:05:17,013 - root - INFO - cluster:C3 node:172.23.107.160:8091 version:7.0.0-5148-enterprise aFamily:inet services:['backup', 'fts', 'index', 'kv', 'n1ql'] 2021-05-11 17:05:17,014 - root - INFO - cluster:C4 node:172.23.107.184:8091 version:7.0.0-5148-enterprise aFamily:inet services:['cbas', 'kv'] 2021-05-11 17:05:17,014 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 17:05:17,014 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 17:05:17,014 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 17:05:17,014 - root - INFO - INSTALL COMPLETED ON: 172.23.105.151 2021-05-11 17:05:17,014 - root - INFO - INSTALL COMPLETED ON: 172.23.105.153 2021-05-11 17:05:17,015 - root - INFO - INSTALL COMPLETED ON: 172.23.107.160 2021-05-11 17:05:17,015 - root - INFO - INSTALL COMPLETED ON: 172.23.107.184 2021-05-11 17:05:17,015 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 17:05:17,015 - root - INFO - TOTAL INSTALL TIME = 304 seconds Running python testrunner.py -i node_conf.ini -c conf/py-multi-node-sanity-1.conf -p get-cbcollect-info=True INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: conf/py-multi-node-sanity-1.conf INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 16 INFO:__main__:--> Running test: xdcr.uniXDCR.unidirectional.load_with_ops,items=5000,ctopology=chain,rdirection=unidirection,update=C1,delete=C1 INFO:__main__:Logs folder: /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_1 *** TestRunner *** {'cluster_name': 'node_conf', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'ini': 'node_conf.ini', 'num_nodes': 4, 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'spec': 'py-multi-node-sanity-1'} Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_1 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t xdcr.uniXDCR.unidirectional.load_with_ops,items=5000,ctopology=chain,rdirection=unidirection,update=C1,delete=C1 Test Input params: {'items': '5000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'update': 'C1', 'delete': 'C1', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 1, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_1'} load_with_ops (xdcr.uniXDCR.unidirectional) ... 2021-05-11 17:05:17,667 - root - ERROR - POST http://172.23.105.151:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2021-05-11 17:05:17,670 - root - INFO - /diag/eval status on 172.23.105.151:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-05-11 17:05:17,690 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:05:17,819 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:05:18,220 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:18,572 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:18,573 - root - INFO - running command.raw on 172.23.105.151: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:18,673 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:18,673 - root - INFO - 0 2021-05-11 17:05:18,675 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:05:18,805 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:05:19,216 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:19,577 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:19,578 - root - INFO - running command.raw on 172.23.105.151: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:19,688 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:19,688 - root - INFO - 0 2021-05-11 17:05:19,691 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:05:19,841 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:05:20,298 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:20,666 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:20,667 - root - INFO - running command.raw on 172.23.105.153: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:20,769 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:20,769 - root - INFO - 0 2021-05-11 17:05:20,772 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:05:20,900 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:05:21,368 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:21,741 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:21,742 - root - INFO - running command.raw on 172.23.105.153: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:21,846 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:21,847 - root - INFO - 0 2021-05-11 17:05:21,849 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:05:21,950 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:05:22,249 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:22,555 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:22,556 - root - INFO - running command.raw on 172.23.107.160: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:22,643 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:22,644 - root - INFO - 0 2021-05-11 17:05:22,647 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:05:22,746 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:05:23,031 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:23,336 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:23,336 - root - INFO - running command.raw on 172.23.107.160: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:23,424 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:23,425 - root - INFO - 0 2021-05-11 17:05:23,427 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:05:23,559 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:05:23,980 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:24,320 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:24,321 - root - INFO - running command.raw on 172.23.107.184: zgrep "panic:" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:24,421 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:24,422 - root - INFO - 0 2021-05-11 17:05:24,423 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:05:24,550 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:05:24,954 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:25,302 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:25,303 - root - INFO - running command.raw on 172.23.107.184: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:25,401 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:25,402 - root - INFO - 0 2021-05-11 17:05:25,402 - root - INFO - {'172.23.105.151': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.105.153': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.160': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.184': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}} 2021-05-11 17:05:25,411 - root - ERROR - POST http://172.23.105.151:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password 2021-05-11 17:05:25,411 - root - INFO - /diag/eval status on 172.23.105.151:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-05-11 17:05:25,413 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:05:25,541 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:05:25,934 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:26,292 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:26,293 - root - INFO - running command.raw on 172.23.105.151: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:26,391 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:26,392 - root - INFO - 0 2021-05-11 17:05:26,394 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:05:26,543 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:05:26,979 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:27,340 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:27,341 - root - INFO - running command.raw on 172.23.105.153: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:27,447 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:27,448 - root - INFO - 0 2021-05-11 17:05:27,450 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:05:27,546 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:05:27,820 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:28,125 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:28,126 - root - INFO - running command.raw on 172.23.107.160: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:28,209 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:28,210 - root - INFO - 0 2021-05-11 17:05:28,212 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:05:28,338 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:05:28,735 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:29,070 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:29,071 - root - INFO - running command.raw on 172.23.107.184: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* 2021-05-11 17:05:29,165 - root - INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... 2021-05-11 17:05:29,166 - root - INFO - 0 2021-05-11 17:05:29,166 - root - INFO - {'172.23.105.151': 0, '172.23.105.153': 0, '172.23.107.160': 0, '172.23.107.184': 0} 2021-05-11 17:05:29,166 - root - INFO - ==== XDCRNewbasetests setup is started for test #1 load_with_ops ==== 2021-05-11 17:05:29,186 - root - INFO - removing xdcr/nodes settings 2021-05-11 17:05:29,211 - root - INFO - cleanup [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root] 2021-05-11 17:05:29,219 - root - INFO - Could not find any buckets for node 172.23.105.151, nothing to delete 2021-05-11 17:05:29,223 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,243 - root - INFO - waiting for ns_server @ 172.23.105.151:8091 2021-05-11 17:05:29,243 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,247 - root - INFO - ns_server @ 172.23.105.151:8091 is running 2021-05-11 17:05:29,255 - root - INFO - Could not find any buckets for node 172.23.105.153, nothing to delete 2021-05-11 17:05:29,260 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,285 - root - INFO - waiting for ns_server @ 172.23.105.153:8091 2021-05-11 17:05:29,285 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,289 - root - INFO - ns_server @ 172.23.105.153:8091 is running 2021-05-11 17:05:29,289 - root - INFO - removing xdcr/nodes settings 2021-05-11 17:05:29,314 - root - INFO - cleanup [ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] 2021-05-11 17:05:29,320 - root - INFO - Could not find any buckets for node 172.23.107.160, nothing to delete 2021-05-11 17:05:29,323 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,343 - root - INFO - waiting for ns_server @ 172.23.107.160:8091 2021-05-11 17:05:29,344 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,347 - root - INFO - ns_server @ 172.23.107.160:8091 is running 2021-05-11 17:05:29,355 - root - INFO - Could not find any buckets for node 172.23.107.184, nothing to delete 2021-05-11 17:05:29,359 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,380 - root - INFO - waiting for ns_server @ 172.23.107.184:8091 2021-05-11 17:05:29,380 - root - INFO - -->is_ns_server_running? 2021-05-11 17:05:29,384 - root - INFO - ns_server @ 172.23.107.184:8091 is running 2021-05-11 17:05:29,384 - root - INFO - Initializing all clusters... 2021-05-11 17:05:29,391 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:05:29,526 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:05:29,958 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:30,334 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:30,351 - root - INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-11 17:05:30,407 - root - INFO - command executed successfully with root 2021-05-11 17:05:30,407 - root - INFO - ['ok'] 2021-05-11 17:05:30,408 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 2021-05-11 17:05:30,410 - root - INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:05:31,175 - root - INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self 2021-05-11 17:05:31,180 - root - INFO - {'uptime': '61', 'memoryTotal': 8192827392, 'memoryFree': 6857838592, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3419, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} 2021-05-11 17:05:31,187 - root - INFO - pools/default params : memoryQuota=4187 2021-05-11 17:05:31,191 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:05:31,192 - root - INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password 2021-05-11 17:05:31,284 - root - INFO - --> status:True 2021-05-11 17:05:31,286 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:05:31,427 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:05:31,819 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:32,176 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:32,187 - root - INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-11 17:05:32,252 - root - INFO - command executed successfully with root 2021-05-11 17:05:32,252 - root - INFO - ['ok'] 2021-05-11 17:05:32,255 - root - INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:05:32,257 - root - INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:05:32,264 - root - INFO - settings/indexes params : storageMode=plasma 2021-05-11 17:05:32,286 - root - INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self 2021-05-11 17:05:32,290 - root - INFO - {'uptime': '82', 'memoryTotal': 8191283200, 'memoryFree': 6981476352, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4186, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} 2021-05-11 17:05:32,297 - root - INFO - pools/default params : memoryQuota=4186 2021-05-11 17:05:32,302 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:05:32,302 - root - INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password 2021-05-11 17:05:32,390 - root - INFO - --> status:True 2021-05-11 17:05:32,392 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:05:32,528 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:05:32,969 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:05:33,335 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:05:33,345 - root - INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-11 17:05:33,410 - root - INFO - command executed successfully with root 2021-05-11 17:05:33,411 - root - INFO - ['ok'] 2021-05-11 17:05:33,414 - root - INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:05:33,416 - root - INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:05:33,422 - root - INFO - settings/indexes params : storageMode=plasma 2021-05-11 17:05:34,443 - root - INFO - adding node 172.23.105.153:8091 to cluster 2021-05-11 17:05:34,444 - root - INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 2021-05-11 17:05:44,461 - root - INFO - rebalance progress took 10.02 seconds 2021-05-11 17:05:44,462 - root - INFO - sleep for 10 seconds after rebalance... 2021-05-11 17:06:04,093 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-05-11 17:06:04,125 - root - INFO - rebalance operation started 2021-05-11 17:06:14,139 - root - INFO - Rebalance - status: none, progress: 100.00% 2021-05-11 17:06:14,146 - root - INFO - rebalancing was completed with progress: 100% in 10.020622730255127 sec 2021-05-11 17:06:14,154 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:06:14,252 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:06:14,555 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:06:14,870 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:06:14,884 - root - INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-11 17:06:15,084 - root - INFO - command executed successfully with root 2021-05-11 17:06:15,084 - root - INFO - ['ok'] 2021-05-11 17:06:15,084 - root - INFO - Enabled diag/eval for non-local hosts from 172.23.107.160 2021-05-11 17:06:15,090 - root - INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:06:15,235 - root - INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self 2021-05-11 17:06:15,241 - root - INFO - {'uptime': '92', 'memoryTotal': 7934996480, 'memoryFree': 6782033920, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3287, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7567, 'curr_items': 0} 2021-05-11 17:06:15,249 - root - INFO - pools/default params : memoryQuota=4055 2021-05-11 17:06:15,261 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:06:15,262 - root - INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password 2021-05-11 17:06:15,359 - root - INFO - --> status:True 2021-05-11 17:06:15,361 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:06:15,472 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:06:15,752 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:06:16,061 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:06:16,071 - root - INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-11 17:06:16,276 - root - INFO - command executed successfully with root 2021-05-11 17:06:16,276 - root - INFO - ['ok'] 2021-05-11 17:06:16,280 - root - INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:06:16,282 - root - INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:06:16,288 - root - INFO - settings/indexes params : storageMode=plasma 2021-05-11 17:06:16,310 - root - INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self 2021-05-11 17:06:16,316 - root - INFO - {'uptime': '92', 'memoryTotal': 8192131072, 'memoryFree': 6505578496, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3163, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['cbas', 'kv'], 'storageTotalRam': 7812, 'curr_items': 0} 2021-05-11 17:06:16,323 - root - INFO - pools/default params : memoryQuota=4187 2021-05-11 17:06:16,328 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-05-11 17:06:16,328 - root - INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password 2021-05-11 17:06:16,420 - root - INFO - --> status:True 2021-05-11 17:06:16,422 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:06:16,549 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:06:16,945 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:06:17,286 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:06:17,296 - root - INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-05-11 17:06:17,354 - root - INFO - command executed successfully with root 2021-05-11 17:06:17,355 - root - INFO - ['ok'] 2021-05-11 17:06:17,360 - root - INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:06:17,362 - root - INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-05-11 17:06:17,367 - root - INFO - settings/indexes params : storageMode=plasma 2021-05-11 17:06:18,383 - root - INFO - adding node 172.23.107.184:8091 to cluster 2021-05-11 17:06:18,383 - root - INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.107.160:8091 2021-05-11 17:06:28,401 - root - INFO - rebalance progress took 10.02 seconds 2021-05-11 17:06:28,401 - root - INFO - sleep for 10 seconds after rebalance... 2021-05-11 17:06:51,718 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-05-11 17:06:51,764 - root - INFO - rebalance operation started 2021-05-11 17:07:01,778 - root - INFO - Rebalance - status: none, progress: 100.00% 2021-05-11 17:07:01,784 - root - INFO - rebalancing was completed with progress: 100% in 10.019363403320312 sec 2021-05-11 17:07:01,784 - root - INFO - Adding user: cbadminbucket with roles: admin 2021-05-11 17:07:01,801 - root - ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-05-11 17:07:01,802 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2021-05-11 17:07:01,934 - root - INFO - Adding user: cbadminbucket with roles: admin 2021-05-11 17:07:01,949 - root - ERROR - DELETE http://172.23.107.160:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-05-11 17:07:01,950 - root - INFO - Exception while deleting user. Exception is -b'"User was not found."' 2021-05-11 17:07:02,219 - root - INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3686&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-11 17:07:02,251 - root - INFO - 0.03 seconds to create bucket default 2021-05-11 17:07:02,252 - root - INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops 2021-05-11 17:07:03,036 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:03,180 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:03,397 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:03,568 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:03,719 - root - WARNING - vbucket map not ready after try 0 2021-05-11 17:07:03,719 - root - INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops 2021-05-11 17:07:03,990 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:04,185 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:04,498 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:04,714 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:04,897 - root - WARNING - vbucket map not ready after try 1 2021-05-11 17:07:04,898 - root - INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops 2021-05-11 17:07:05,140 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:05,227 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:05,339 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:05,437 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:05,497 - root - WARNING - vbucket map not ready after try 2 2021-05-11 17:07:05,497 - root - INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops 2021-05-11 17:07:05,627 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:05,698 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:05,818 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:05,885 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:05,949 - root - WARNING - vbucket map not ready after try 3 2021-05-11 17:07:05,949 - root - INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops 2021-05-11 17:07:06,064 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:06,135 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:06,259 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:06,331 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:06,399 - root - WARNING - vbucket map not ready after try 4 2021-05-11 17:07:06,400 - root - INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops 2021-05-11 17:07:06,530 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:06,594 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:06,722 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:06,800 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:06,881 - root - WARNING - vbucket map not ready after try 5 2021-05-11 17:07:06,916 - root - INFO - http://172.23.105.151:8091/pools/default/buckets/default/scopes with params: name=scope_1 2021-05-11 17:07:06,917 - root - INFO - Scope created default->scope_1 {'uid': '1'} 2021-05-11 17:07:06,951 - root - INFO - http://172.23.105.151:8091/pools/default/buckets/default/scopes/scope_1/collections with params: name=collection_1 2021-05-11 17:07:06,952 - root - INFO - Collection created default->scope_1->collection_1 manifest:{'uid': '2'} 2021-05-11 17:07:07,821 - root - INFO - http://172.23.107.160:8091/pools/default/buckets with param: name=default&ramQuotaMB=3555&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-05-11 17:07:07,853 - root - INFO - 0.03 seconds to create bucket default 2021-05-11 17:07:07,853 - root - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops 2021-05-11 17:07:08,614 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:08,757 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:09,003 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:09,157 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:09,286 - root - WARNING - vbucket map not ready after try 0 2021-05-11 17:07:09,287 - root - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops 2021-05-11 17:07:09,547 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:09,754 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:10,063 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:10,234 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:10,364 - root - WARNING - vbucket map not ready after try 1 2021-05-11 17:07:10,365 - root - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops 2021-05-11 17:07:10,673 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:10,855 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:11,069 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:11,208 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:11,324 - root - WARNING - vbucket map not ready after try 2 2021-05-11 17:07:11,324 - root - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops 2021-05-11 17:07:11,551 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:11,692 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:11,926 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:12,073 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:12,209 - root - WARNING - vbucket map not ready after try 3 2021-05-11 17:07:12,209 - root - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops 2021-05-11 17:07:12,421 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:12,549 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:12,775 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:12,901 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:13,024 - root - WARNING - vbucket map not ready after try 4 2021-05-11 17:07:13,024 - root - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops 2021-05-11 17:07:13,239 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:13,372 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:13,602 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:13,727 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:13,866 - root - WARNING - vbucket map not ready after try 5 2021-05-11 17:07:13,901 - root - INFO - http://172.23.107.160:8091/pools/default/buckets/default/scopes with params: name=scope_1 2021-05-11 17:07:13,902 - root - INFO - Scope created default->scope_1 {'uid': '1'} 2021-05-11 17:07:13,936 - root - INFO - http://172.23.107.160:8091/pools/default/buckets/default/scopes/scope_1/collections with params: name=collection_1 2021-05-11 17:07:13,936 - root - INFO - Collection created default->scope_1->collection_1 manifest:{'uid': '2'} 2021-05-11 17:07:13,936 - root - INFO - ==== XDCRNewbasetests setup is finished for test #1 load_with_ops ==== 2021-05-11 17:07:13,943 - root - INFO - adding remote cluster hostname:172.23.107.160:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.105.151:8091 2021-05-11 17:07:24,014 - root - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {} 2021-05-11 17:07:24,060 - root - INFO - Replication created with id: 4a6a3cd6caa13f37f6170006d2a50e7d/default/default 2021-05-11 17:07:34,094 - root - INFO - Updated checkpointInterval=60 on bucket 'default' on 172.23.105.151 2021-05-11 17:07:34,152 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:34,223 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:34,982 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:34,987 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:34,989 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:34,990 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:35,049 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:35,062 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:35,064 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:35,065 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:35,343 - root - INFO - Updating keys @ C1 2021-05-11 17:07:35,344 - root - INFO - At bucket 'default' @ C1: operation: update, key range 0 - 300 2021-05-11 17:07:35,420 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:35,517 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:36,034 - root - INFO - Deleting keys @ C1 2021-05-11 17:07:36,034 - root - INFO - At bucket 'default' @ C1: operation: delete, key range 700 - 1000 2021-05-11 17:07:36,100 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:36,177 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:37,175 - root - INFO - Merging keys for replication Replication C1:default -> C2:default 2021-05-11 17:07:37,196 - root - INFO - src_kvstore has 700 valid and 300 deleted keys 2021-05-11 17:07:37,196 - root - INFO - dest kvstore has 0 valid and 0 deleted keys 2021-05-11 17:07:37,219 - root - INFO - After merging: destination bucket's kv_store now has 700 valid keys and 0 deleted keys 2021-05-11 17:07:37,253 - root - INFO - dict:{'ip': '172.23.105.151', 'port': '8091', 'username': 'Administrator', 'password': 'password'} 2021-05-11 17:07:37,254 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:37,320 - root - INFO - Setting flush param on server {'ip': '172.23.105.151', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default 2021-05-11 17:07:37,320 - root - INFO - setting param: exp_pager_stime 10 2021-05-11 17:07:37,321 - root - INFO - Setting flush param on server {'ip': '172.23.105.151', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (2679510786, 0, b'') 2021-05-11 17:07:37,344 - root - INFO - dict:{'ip': '172.23.105.153', 'port': '8091', 'username': 'Administrator', 'password': 'password'} 2021-05-11 17:07:37,344 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:37,418 - root - INFO - Setting flush param on server {'ip': '172.23.105.153', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default 2021-05-11 17:07:37,418 - root - INFO - setting param: exp_pager_stime 10 2021-05-11 17:07:37,421 - root - INFO - Setting flush param on server {'ip': '172.23.105.153', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (939689406, 0, b'') 2021-05-11 17:07:37,421 - root - INFO - wait for expiry pager to run on all these nodes 2021-05-11 17:07:47,461 - root - INFO - dict:{'ip': '172.23.107.160', 'port': '8091', 'username': 'Administrator', 'password': 'password'} 2021-05-11 17:07:47,461 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:47,542 - root - INFO - Setting flush param on server {'ip': '172.23.107.160', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default 2021-05-11 17:07:47,542 - root - INFO - setting param: exp_pager_stime 10 2021-05-11 17:07:47,543 - root - INFO - Setting flush param on server {'ip': '172.23.107.160', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (3971287937, 0, b'') 2021-05-11 17:07:47,563 - root - INFO - dict:{'ip': '172.23.107.184', 'port': '8091', 'username': 'Administrator', 'password': 'password'} 2021-05-11 17:07:47,564 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:47,620 - root - INFO - Setting flush param on server {'ip': '172.23.107.184', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default 2021-05-11 17:07:47,620 - root - INFO - setting param: exp_pager_stime 10 2021-05-11 17:07:47,621 - root - INFO - Setting flush param on server {'ip': '172.23.107.184', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (2560813875, 0, b'') 2021-05-11 17:07:47,621 - root - INFO - wait for expiry pager to run on all these nodes 2021-05-11 17:07:58,001 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:07:58,116 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket 2021-05-11 17:07:58,146 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:07:58,296 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.153:8091',default bucket 2021-05-11 17:07:58,940 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:07:59,049 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.160:8091',default bucket 2021-05-11 17:07:59,078 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:07:59,219 - root - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.184:8091',default bucket 2021-05-11 17:07:59,220 - root - INFO - Waiting for dcp queue to drain on cluster node: 172.23.105.151 2021-05-11 17:07:59,229 - root - INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:07:59,975 - root - INFO - Current dcp queue size on C1 for default is 0 2021-05-11 17:07:59,975 - root - INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.160 2021-05-11 17:07:59,982 - root - INFO - http://172.23.107.160:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:08:00,697 - root - ERROR - 'ep_dcp_xdcr_items_remaining' 2021-05-11 17:08:00,891 - root - INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 2021-05-11 17:08:00,960 - root - WARNING - Stat "replication_changes_left" not found 2021-05-11 17:08:00,960 - root - ERROR - local variable 'mutations' referenced before assignment 2021-05-11 17:08:00,968 - root - INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:08:01,656 - root - INFO - Saw: vb_active_curr_items 700 == 700 expected on C1, default bucket 2021-05-11 17:08:01,659 - root - INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:08:02,342 - root - INFO - Saw: vb_replica_curr_items 700 == 700 expected on C1, default bucket 2021-05-11 17:08:02,356 - root - INFO - http://172.23.107.160:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:08:03,037 - root - INFO - Saw: vb_active_curr_items 700 == 700 expected on C2, default bucket 2021-05-11 17:08:03,040 - root - INFO - http://172.23.107.160:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:08:03,716 - root - INFO - Saw: vb_replica_curr_items 700 == 700 expected on C2, default bucket 2021-05-11 17:08:03,784 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:08:03,849 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:08:03,940 - root - INFO - 1000 items will be verified on default bucket 2021-05-11 17:08:04,304 - root - INFO - 0 items were verified 2021-05-11 17:08:04,801 - root - INFO - 1000 items were verified in 0.8602104187011719 sec.the average number of ops - 1162.0 per second 2021-05-11 17:08:04,906 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:08:04,981 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:08:05,065 - root - INFO - 700 items will be verified on default bucket 2021-05-11 17:08:05,227 - root - INFO - 0 items were verified 2021-05-11 17:08:05,495 - root - INFO - 700 items were verified in 0.42946934700012207 sec.the average number of ops - 1629.0 per second 2021-05-11 17:08:05,496 - root - INFO - Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.105.151 -> Couchbase Cluster: C2, Master Ip: 172.23.107.160, bucket default 2021-05-11 17:08:05,556 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:08:05,664 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:08:05,827 - root - INFO - creating direct client 172.23.105.151:11210 default 2021-05-11 17:08:05,937 - root - INFO - creating direct client 172.23.105.153:11210 default 2021-05-11 17:08:06,057 - root - INFO - creating direct client 172.23.107.160:11210 default 2021-05-11 17:08:06,154 - root - INFO - creating direct client 172.23.107.184:11210 default 2021-05-11 17:08:06,243 - root - INFO - RevID verification: in progress for default in scope:None in collection: None 2021-05-11 17:08:08,324 - root - INFO - RevId Verification : 700 existing items have been verified 2021-05-11 17:08:08,324 - root - INFO - RevId Verification : 300 deleted items have been verified 2021-05-11 17:08:08,324 - root - INFO - RevId Verification : 0 keys were apparently filtered out and not found in target bucket 2021-05-11 17:08:08,329 - root - INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute 2021-05-11 17:08:09,077 - root - INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-05-11 17:08:09,081 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:08:09,212 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:08:09,618 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:09,973 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:09,975 - root - INFO - running command.raw on 172.23.105.151: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:10,053 - root - INFO - command executed successfully with root 2021-05-11 17:08:10,054 - root - INFO - 0 2021-05-11 17:08:10,054 - root - INFO - [] 2021-05-11 17:08:10,054 - root - INFO - Initial replication restart count on 172.23.105.151 :0, now :0 2021-05-11 17:08:10,056 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:08:10,200 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:08:10,648 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:11,013 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:11,014 - root - INFO - running command.raw on 172.23.105.153: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:11,089 - root - INFO - command executed successfully with root 2021-05-11 17:08:11,089 - root - INFO - 0 2021-05-11 17:08:11,089 - root - INFO - [] 2021-05-11 17:08:11,089 - root - INFO - Initial replication restart count on 172.23.105.153 :0, now :0 2021-05-11 17:08:11,091 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:08:11,200 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:08:11,526 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:11,842 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:11,843 - root - INFO - running command.raw on 172.23.107.160: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:11,910 - root - INFO - command executed successfully with root 2021-05-11 17:08:11,910 - root - INFO - 0 2021-05-11 17:08:11,910 - root - INFO - [] 2021-05-11 17:08:11,910 - root - INFO - Initial replication restart count on 172.23.107.160 :0, now :0 2021-05-11 17:08:11,912 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:08:12,119 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:08:12,599 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:13,011 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:13,012 - root - INFO - running command.raw on 172.23.107.184: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:13,088 - root - INFO - command executed successfully with root 2021-05-11 17:08:13,089 - root - INFO - 0 2021-05-11 17:08:13,089 - root - INFO - [] 2021-05-11 17:08:13,089 - root - INFO - Initial replication restart count on 172.23.107.184 :0, now :0 2021-05-11 17:08:13,114 - root - INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-05-11 17:08:13,117 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:08:13,282 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:08:13,734 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:14,162 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:14,163 - root - INFO - running command.raw on 172.23.105.151: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:14,208 - root - INFO - command executed successfully with root 2021-05-11 17:08:14,209 - root - INFO - 0 2021-05-11 17:08:14,209 - root - INFO - Initial panic: count on 172.23.105.151 :0, now :0 2021-05-11 17:08:14,211 - root - INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 2021-05-11 17:08:14,369 - root - INFO - SSH Connected to 172.23.105.151 as root 2021-05-11 17:08:14,806 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:15,232 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:15,233 - root - INFO - running command.raw on 172.23.105.151: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:15,271 - root - INFO - command executed successfully with root 2021-05-11 17:08:15,271 - root - INFO - 0 2021-05-11 17:08:15,272 - root - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.151 :0, now :0 2021-05-11 17:08:15,274 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:08:15,443 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:08:15,963 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:16,417 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:16,418 - root - INFO - running command.raw on 172.23.105.153: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:16,501 - root - INFO - command executed successfully with root 2021-05-11 17:08:16,501 - root - INFO - 0 2021-05-11 17:08:16,501 - root - INFO - Initial panic: count on 172.23.105.153 :0, now :0 2021-05-11 17:08:16,503 - root - INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 2021-05-11 17:08:16,676 - root - INFO - SSH Connected to 172.23.105.153 as root 2021-05-11 17:08:17,195 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:17,643 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:17,644 - root - INFO - running command.raw on 172.23.105.153: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:17,727 - root - INFO - command executed successfully with root 2021-05-11 17:08:17,728 - root - INFO - 0 2021-05-11 17:08:17,728 - root - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.153 :0, now :0 2021-05-11 17:08:17,730 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:08:17,900 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:08:18,259 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:18,643 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:18,644 - root - INFO - running command.raw on 172.23.107.160: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:18,675 - root - INFO - command executed successfully with root 2021-05-11 17:08:18,675 - root - INFO - 0 2021-05-11 17:08:18,676 - root - INFO - Initial panic: count on 172.23.107.160 :0, now :0 2021-05-11 17:08:18,678 - root - INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 2021-05-11 17:08:18,846 - root - INFO - SSH Connected to 172.23.107.160 as root 2021-05-11 17:08:19,200 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:19,585 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:19,587 - root - INFO - running command.raw on 172.23.107.160: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:19,653 - root - INFO - command executed successfully with root 2021-05-11 17:08:19,654 - root - INFO - 0 2021-05-11 17:08:19,654 - root - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.160 :0, now :0 2021-05-11 17:08:19,656 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:08:19,820 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:08:20,288 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:20,715 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:20,716 - root - INFO - running command.raw on 172.23.107.184: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:20,754 - root - INFO - command executed successfully with root 2021-05-11 17:08:20,755 - root - INFO - 0 2021-05-11 17:08:20,755 - root - INFO - Initial panic: count on 172.23.107.184 :0, now :0 2021-05-11 17:08:20,758 - root - INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 2021-05-11 17:08:20,959 - root - INFO - SSH Connected to 172.23.107.184 as root 2021-05-11 17:08:21,421 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-05-11 17:08:21,847 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-05-11 17:08:21,847 - root - INFO - running command.raw on 172.23.107.184: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* 2021-05-11 17:08:21,887 - root - INFO - command executed successfully with root 2021-05-11 17:08:21,888 - root - INFO - 0 2021-05-11 17:08:21,888 - root - INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.184 :0, now :0 2021-05-11 17:08:21,981 - root - INFO - ==== XDCRNewbasetests cleanup is started for test #1 load_with_ops ==== 2021-05-11 17:08:21,981 - root - INFO - removing xdcr/nodes settings 2021-05-11 17:08:22,014 - root - INFO - Deleting replication /controller/cancelXDCR/4a6a3cd6caa13f37f6170006d2a50e7d%2Fdefault%2Fdefault 2021-05-11 17:08:22,059 - root - INFO - Replication deleted successfully 2021-05-11 17:08:22,077 - root - INFO - removing remote cluster name:remote_cluster_C1-C2 2021-05-11 17:08:22,132 - root - INFO - cleanup [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root] 2021-05-11 17:08:22,175 - root - INFO - deleting existing buckets ['default'] on 172.23.105.151 2021-05-11 17:08:23,108 - root - INFO - waiting for bucket deletion to complete.... 2021-05-11 17:08:23,121 - root - INFO - node 172.23.105.151 existing buckets : [] 2021-05-11 17:08:23,121 - root - INFO - deleted bucket : default from 172.23.105.151 2021-05-11 17:08:23,136 - root - INFO - -->is_ns_server_running? 2021-05-11 17:08:23,178 - root - INFO - rebalancing all nodes in order to remove nodes 2021-05-11 17:08:23,190 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} 2021-05-11 17:08:23,233 - root - INFO - rebalance operation started 2021-05-11 17:08:23,244 - root - INFO - rebalance percentage : 0.00 % 2021-05-11 17:08:43,289 - root - INFO - rebalance progress took 20.06 seconds 2021-05-11 17:08:43,290 - root - INFO - sleep for 10 seconds after rebalance... 2021-05-11 17:08:53,327 - root - INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] 2021-05-11 17:08:53,335 - root - INFO - waiting for ns_server @ 172.23.105.151:8091 2021-05-11 17:08:53,335 - root - INFO - -->is_ns_server_running? 2021-05-11 17:08:53,339 - root - INFO - ns_server @ 172.23.105.151:8091 is running 2021-05-11 17:08:53,364 - root - INFO - Could not find any buckets for node 172.23.105.153, nothing to delete 2021-05-11 17:08:53,377 - root - INFO - -->is_ns_server_running? 2021-05-11 17:08:53,430 - root - INFO - waiting for ns_server @ 172.23.105.153:8091 2021-05-11 17:08:53,430 - root - INFO - -->is_ns_server_running? 2021-05-11 17:08:53,444 - root - INFO - ns_server @ 172.23.105.153:8091 is running Cluster instance shutdown with force 2021-05-11 17:08:53,444 - root - INFO - removing xdcr/nodes settings 2021-05-11 17:08:53,468 - root - INFO - cleanup [ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] 2021-05-11 17:08:53,486 - root - INFO - deleting existing buckets ['default'] on 172.23.107.160 2021-05-11 17:08:54,686 - root - INFO - waiting for bucket deletion to complete.... 2021-05-11 17:08:54,690 - root - INFO - node 172.23.107.160 existing buckets : [] 2021-05-11 17:08:54,690 - root - INFO - deleted bucket : default from 172.23.107.160 2021-05-11 17:08:54,698 - root - INFO - -->is_ns_server_running? 2021-05-11 17:08:54,711 - root - INFO - rebalancing all nodes in order to remove nodes 2021-05-11 17:08:54,715 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.107.184', 'user': 'Administrator', 'password': 'password'} 2021-05-11 17:08:54,765 - root - INFO - rebalance operation started 2021-05-11 17:08:54,774 - root - INFO - rebalance percentage : 0.00 % 2021-05-11 17:09:14,817 - root - INFO - rebalance progress took 20.05 seconds 2021-05-11 17:09:14,817 - root - INFO - sleep for 10 seconds after rebalance... 2021-05-11 17:09:24,850 - root - INFO - removed all the nodes from cluster associated with ip:172.23.107.160 port:8091 ssh_username:root ? [('ns_1@172.23.107.184', 8091)] 2021-05-11 17:09:24,866 - root - INFO - waiting for ns_server @ 172.23.107.160:8091 2021-05-11 17:09:24,866 - root - INFO - -->is_ns_server_running? 2021-05-11 17:09:24,881 - root - INFO - ns_server @ 172.23.107.160:8091 is running 2021-05-11 17:09:24,908 - root - INFO - Could not find any buckets for node 172.23.107.184, nothing to delete 2021-05-11 17:09:24,923 - root - INFO - -->is_ns_server_running? 2021-05-11 17:09:24,959 - root - INFO - waiting for ns_server @ 172.23.107.184:8091 2021-05-11 17:09:24,959 - root - INFO - -->is_ns_server_running? 2021-05-11 17:09:24,967 - root - INFO - ns_server @ 172.23.107.184:8091 is running Cluster instance shutdown with force 2021-05-11 17:09:24,967 - root - INFO - ==== XDCRNewbasetests cleanup is finished for test #1 load_with_ops === Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_1 ok ---------------------------------------------------------------------- Ran 1 test in 247.311s OK load_with_async_ops (xdcr.biXDCR.bidirectional) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_2 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t xdcr.biXDCR.bidirectional.load_with_async_ops,replicas=1,items=1000,ctopology=chain,rdirection=bidirection,update=C1-C2,delete=C1-C2 Test Input params: {'replicas': '1', 'items': '1000', 'ctopology': 'chain', 'rdirection': 'bidirection', 'update': 'C1-C2', 'delete': 'C1-C2', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 2, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_2'} [2021-05-11 17:09:24,997] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 17:09:24,999] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:09:25,144] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:09:25,610] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:26,045] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:26,046] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:26,086] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:26,086] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:26,088] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:09:26,212] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:09:26,622] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:26,988] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:26,989] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:27,014] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:27,014] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:27,016] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:09:27,153] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:09:27,617] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:27,996] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:27,997] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:28,067] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:28,068] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:28,069] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:09:28,201] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:09:28,668] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:29,052] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:29,053] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:29,125] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:29,126] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:29,128] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:09:29,229] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:09:29,545] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:29,875] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:29,876] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:29,936] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:29,937] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:29,939] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:09:30,035] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:09:30,323] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:30,638] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:30,639] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:30,703] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:30,703] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:30,705] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:09:30,838] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:09:31,241] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:31,586] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:31,587] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:31,652] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:31,653] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:31,656] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:09:31,783] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:09:32,188] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:32,527] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:32,527] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:32,592] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:32,593] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:32,593] - [xdcrnewbasetests:2981] INFO - {'172.23.105.151': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.105.153': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.160': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.184': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}} [2021-05-11 17:09:32,621] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 17:09:32,623] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:09:32,757] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:09:33,156] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:33,506] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:33,507] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:33,577] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:33,578] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:33,580] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:09:33,725] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:09:34,205] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:34,592] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:34,593] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:34,667] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:34,667] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:34,669] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:09:34,768] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:09:35,058] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:35,369] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:35,370] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:35,390] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:35,391] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:35,393] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:09:35,532] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:09:35,953] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:36,301] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:36,302] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 17:09:36,369] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:36,370] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 17:09:36,370] - [xdcrnewbasetests:2999] INFO - {'172.23.105.151': 0, '172.23.105.153': 0, '172.23.107.160': 0, '172.23.107.184': 0} [2021-05-11 17:09:36,370] - [xdcrnewbasetests:2741] INFO - ==== XDCRNewbasetests setup is started for test #2 load_with_async_ops ==== [2021-05-11 17:09:36,393] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 17:09:36,416] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root] [2021-05-11 17:09:36,423] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:09:36,427] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,442] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:09:36,443] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,447] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:09:36,457] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:09:36,462] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,478] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:09:36,479] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,482] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:09:36,482] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 17:09:36,503] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:09:36,510] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:09:36,514] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,529] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:09:36,529] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,533] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:09:36,540] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:09:36,543] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,575] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:09:36,575] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:09:36,588] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:09:36,589] - [xdcrnewbasetests:3009] INFO - Initializing all clusters... [2021-05-11 17:09:36,605] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:09:36,773] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:09:37,237] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:37,689] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:37,716] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:09:37,776] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:37,777] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:09:37,777] - [xdcrnewbasetests:1239] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:09:37,787] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:09:38,386] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:09:38,399] - [task:157] INFO - {'uptime': '306', 'memoryTotal': 8192827392, 'memoryFree': 6743519232, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 17:09:38,415] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:09:38,435] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:09:38,435] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:09:38,537] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:09:38,541] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:09:38,706] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:09:39,186] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:39,642] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:39,668] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:09:39,733] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:39,734] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:09:39,744] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:09:39,754] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:09:39,768] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:09:39,821] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:09:39,832] - [task:157] INFO - {'uptime': '333', 'memoryTotal': 8191283200, 'memoryFree': 6989676544, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:09:39,843] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:09:39,844] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:09:39,858] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:09:39,859] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:09:39,961] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:09:39,963] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:09:40,122] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:09:40,625] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:09:41,065] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:09:41,092] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:09:41,156] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:09:41,156] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:09:41,168] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:09:41,178] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:09:41,192] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:09:42,233] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 17:09:42,233] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:09:52,251] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 17:09:52,251] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:10:11,963] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/nodeStatuses body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password [2021-05-11 17:10:11,971] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/pools/default error [Errno 104] Connection reset by peer [2021-05-11 17:10:15,000] - [rest_client:1727] INFO - rebalance params : {'knownNodes': '', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:10:15,012] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/controller/rebalance body: knownNodes=&ejectedNodes=&user=Administrator&password=password headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"empty_known_nodes":1}' auth: Administrator:password [2021-05-11 17:10:15,013] - [rest_client:1734] ERROR - rebalance operation failed: b'{"empty_known_nodes":1}' [, , , , , ] Tue May 11 17:10:15 2021 [, , , , , , , , , , , , ] [2021-05-11 17:10:15,017] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:10:15,019] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:10:15,022] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:10:15,025] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:10:15,188] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:10:15,196] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:10:15,203] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:10:15,204] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:10:15,583] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:10:15,674] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:10:15,694] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:10:15,734] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:10:15,970] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.160 [2021-05-11 17:10:15,970] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/cbcollect_info 172.23.107.160-20210511-1710-diag.zip [2021-05-11 17:10:16,110] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.151 [2021-05-11 17:10:16,111] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbcollect_info 172.23.105.151-20210511-1710-diag.zip [2021-05-11 17:10:16,113] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.184 [2021-05-11 17:10:16,114] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/cbcollect_info 172.23.107.184-20210511-1710-diag.zip [2021-05-11 17:10:16,192] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.153 [2021-05-11 17:10:16,193] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/cbcollect_info 172.23.105.153-20210511-1710-diag.zip [2021-05-11 17:12:52,996] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.105.153 [2021-05-11 17:12:54,650] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -f /root/172.23.105.153-20210511-1710-diag.zip [2021-05-11 17:12:54,689] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:12:58,427] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.107.184 [2021-05-11 17:12:59,827] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -f /root/172.23.107.184-20210511-1710-diag.zip [2021-05-11 17:12:59,863] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:14:55,420] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.107.160 [2021-05-11 17:14:57,047] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -f /root/172.23.107.160-20210511-1710-diag.zip [2021-05-11 17:14:57,083] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:15:02,436] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.105.151 [2021-05-11 17:15:03,587] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -f /root/172.23.105.151-20210511-1710-diag.zip [2021-05-11 17:15:03,628] - [remote_util:3475] INFO - command executed successfully with root grabbing dbinfo from 172.23.105.151 [2021-05-11 17:15:03,631] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:15:03,778] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:15:04,194] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:15:04,565] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.105.151 [2021-05-11 17:15:04,567] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.105.151-20210511-1715-couch-dbinfo.txt [2021-05-11 17:15:04,622] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:15:04,623] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file [2021-05-11 17:15:04,718] - [remote_util:1500] INFO - File 172.23.105.151-20210511-1715-couch-dbinfo.txt will be deleted [2021-05-11 17:15:04,718] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /root/*172.23.105.151-20210511-1715-couch-dbinfo.txt* [2021-05-11 17:15:04,747] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:15:04,748] - [remote_util:338] INFO - 172.23.105.151:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.105.153 [2021-05-11 17:15:34,775] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:15:34,921] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:15:35,392] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:15:35,767] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.105.153 [2021-05-11 17:15:35,767] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.105.153-20210511-1715-couch-dbinfo.txt [2021-05-11 17:15:35,820] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:15:35,820] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file [2021-05-11 17:15:35,916] - [remote_util:1500] INFO - File 172.23.105.153-20210511-1715-couch-dbinfo.txt will be deleted [2021-05-11 17:15:35,916] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /root/*172.23.105.153-20210511-1715-couch-dbinfo.txt* [2021-05-11 17:15:35,939] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:15:35,939] - [remote_util:338] INFO - 172.23.105.153:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.107.160 [2021-05-11 17:16:05,972] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:16:06,077] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:16:06,375] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:16:06,702] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.107.160 [2021-05-11 17:16:06,703] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.160-20210511-1716-couch-dbinfo.txt [2021-05-11 17:16:06,737] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:16:06,738] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file [2021-05-11 17:16:06,807] - [remote_util:1500] INFO - File 172.23.107.160-20210511-1716-couch-dbinfo.txt will be deleted [2021-05-11 17:16:06,808] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /root/*172.23.107.160-20210511-1716-couch-dbinfo.txt* [2021-05-11 17:16:06,824] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:16:06,824] - [remote_util:338] INFO - 172.23.107.160:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.107.184 [2021-05-11 17:16:36,857] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:16:37,027] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:16:37,531] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:16:37,959] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.107.184 [2021-05-11 17:16:37,960] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.184-20210511-1716-couch-dbinfo.txt [2021-05-11 17:16:38,018] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:16:38,019] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0": no header in non-empty file Failed to open "/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0": no header in non-empty file [2021-05-11 17:16:38,173] - [remote_util:1500] INFO - File 172.23.107.184-20210511-1716-couch-dbinfo.txt will be deleted [2021-05-11 17:16:38,173] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /root/*172.23.107.184-20210511-1716-couch-dbinfo.txt* [2021-05-11 17:16:38,203] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:16:38,203] - [remote_util:338] INFO - 172.23.107.184:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_2 Traceback (most recent call last): File "lib/tasks/task.py", line 756, in execute self.start_rebalance(task_manager) File "lib/tasks/task.py", line 815, in start_rebalance self.rest.rebalance(otpNodes=[node.id for node in nodes], ejectedNodes=ejectedNodes) File "lib/membase/api/rest_client.py", line 1737, in rebalance parameters=params) membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password ERROR ====================================================================== ERROR: load_with_async_ops (xdcr.biXDCR.bidirectional) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/xdcr/biXDCR.py", line 16, in setUp super(bidirectional, self).setUp() File "pytests/xdcr/xdcrnewbasetests.py", line 2743, in setUp self.__setup_for_test() File "pytests/xdcr/xdcrnewbasetests.py", line 2862, in __setup_for_test self.__init_clusters() File "pytests/xdcr/xdcrnewbasetests.py", line 3014, in __init_clusters cluster.init_cluster(disabled_consistent_view) File "pytests/xdcr/xdcrnewbasetests.py", line 1211, in init_cluster use_hostnames=self.__use_hostname).result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception File "lib/tasks/task.py", line 756, in execute self.start_rebalance(task_manager) File "lib/tasks/task.py", line 815, in start_rebalance self.rest.rebalance(otpNodes=[node.id for node in nodes], ejectedNodes=ejectedNodes) File "lib/membase/api/rest_client.py", line 1737, in rebalance parameters=params) membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password ---------------------------------------------------------------------- Ran 1 test in 50.036s FAILED (errors=1) rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_3 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=1000,doc_ops=create;update;delete Test Input params: {'nodes_in': '3', 'replicas': '1', 'items': '1000', 'doc_ops': 'create;update;delete', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 3, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_3'} [2021-05-11 17:17:08,255] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:17:08,393] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:17:08,786] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:09,154] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:09,171] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:17:09,213] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:17:09,237] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveAdded [2021-05-11 17:17:09,237] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:17:09,237] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #3 rebalance_in_with_ops============== [2021-05-11 17:17:09,287] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:17:09,297] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:17:09,319] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:17:09,339] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:17:09,351] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:17:09,385] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 17:17:09,396] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:17:09,434] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:17:09,444] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:17:29,490] - [rest_client:1833] INFO - rebalance progress took 20.06 seconds [2021-05-11 17:17:29,490] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:17:39,515] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 17:17:39,528] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:17:39,528] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:17:39,541] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:17:39,546] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:17:39,546] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:17:39,551] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:17:39,565] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:17:39,566] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:17:39,578] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:17:39,592] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:17:39,592] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:17:39,604] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:17:39,605] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:17:40,303] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:17:40,317] - [task:157] INFO - {'uptime': '788', 'memoryTotal': 8192827392, 'memoryFree': 6763601920, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 17:17:40,334] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 17:17:40,334] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:17:40,334] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:17:40,351] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 17:17:40,351] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:17:40,351] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:17:40,368] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 17:17:40,385] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 17:17:40,385] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 17:17:40,398] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:17:40,398] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:17:40,398] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:17:40,398] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:17:40,499] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:17:40,501] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:17:40,664] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:17:41,145] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:41,616] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:41,649] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:17:41,708] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:17:41,709] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:17:41,718] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:41,727] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:41,741] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:17:41,791] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:17:41,802] - [task:157] INFO - {'uptime': '813', 'memoryTotal': 8191283200, 'memoryFree': 6985498624, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:17:41,812] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:17:41,812] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:17:41,824] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:17:41,824] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:17:41,929] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:17:41,931] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:17:42,107] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:17:42,674] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:43,112] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:43,143] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:17:43,207] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:17:43,208] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:17:43,219] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:43,229] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:43,242] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:17:43,294] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:17:43,306] - [task:157] INFO - {'uptime': '778', 'memoryTotal': 7934996480, 'memoryFree': 6707625984, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4055, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:17:43,319] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:17:43,332] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:17:43,332] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:17:43,431] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:17:43,433] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:17:43,612] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:17:43,987] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:44,381] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:44,406] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:17:44,607] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:17:44,608] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:17:44,619] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:44,630] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:44,644] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:17:44,697] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:17:44,710] - [task:157] INFO - {'uptime': '780', 'memoryTotal': 8192131072, 'memoryFree': 6991798272, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:17:44,722] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:17:44,723] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:17:44,736] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:17:44,737] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:17:44,839] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:17:44,841] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:17:45,016] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:17:45,523] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:45,962] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:45,994] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:17:46,046] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:17:46,046] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:17:46,056] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:46,064] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:17:46,075] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:17:46,126] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:17:46,151] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 17:17:46,153] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 17:17:46,260] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:17:51,265] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:17:51,281] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:17:51,284] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:17:51,437] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:17:51,919] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:52,371] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:52,386] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:17:52,466] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:17:52,467] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:17:52,467] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:17:53,143] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3419&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:17:53,178] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 17:17:53,179] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:17:53,944] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:54,270] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:54,395] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:17:54,395] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:17:54,658] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:54,924] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:55,098] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:17:55,098] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:17:55,317] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:55,561] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:55,709] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:17:55,709] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:17:55,972] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:56,218] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:56,412] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:17:56,413] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:17:56,699] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:57,258] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:57,433] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:17:57,433] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:17:57,770] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:58,087] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:17:58,273] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:17:58,275] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #3 rebalance_in_with_ops ============== [2021-05-11 17:17:58,306] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:17:58,472] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:17:59,151] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:17:59,569] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:17:59,571] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:17:59,757] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:18:00,325] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:18:00,780] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:18:00,783] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:18:00,966] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:18:01,390] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:18:01,790] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:18:01,795] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:18:02,013] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:18:02,536] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:18:02,960] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:18:10,295] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:18:10,296] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.56852791878173, 'mem_free': 6668480512, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:18:10,296] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:18:10,340] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:18:11,444] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:18:11,445] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:18:11,446] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:18:11,446] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:18:11,780] - [basetestcase:3022] INFO - Servers to check bucket-seqno: [ip:172.23.105.151 port:8091 ssh_username:root] [2021-05-11 17:18:11,780] - [basetestcase:2981] INFO - Record vbucket for the bucket default [2021-05-11 17:18:11,823] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:18:13,293] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 17:18:13,293] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:18:23,306] - [rest_client:1833] INFO - rebalance progress took 10.01 seconds [2021-05-11 17:18:23,306] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:18:42,389] - [task:769] INFO - adding node 172.23.107.160:8091 to cluster [2021-05-11 17:18:42,389] - [rest_client:1500] INFO - adding remote node @172.23.107.160:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:18:52,409] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 17:18:52,409] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:19:06,356] - [task:769] INFO - adding node 172.23.107.184:8091 to cluster [2021-05-11 17:19:06,356] - [rest_client:1500] INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:19:16,381] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 17:19:16,381] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:19:35,225] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:19:35,284] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:19:45,299] - [rest_client:1894] INFO - rebalance percentage : 8.00 % [2021-05-11 17:19:45,299] - [task:839] INFO - Rebalance - status: running, progress: 8.00% [2021-05-11 17:20:05,356] - [rest_client:1894] INFO - rebalance percentage : 62.00 % [2021-05-11 17:20:05,356] - [task:839] INFO - Rebalance - status: running, progress: 62.00% [2021-05-11 17:20:25,419] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 17:20:25,431] - [task:898] INFO - rebalancing was completed with progress: 100% in 50.14617109298706 sec [2021-05-11 17:20:25,431] - [basetestcase:662] INFO - sleep for 10 secs. Sleep before stat verification ... [2021-05-11 17:20:35,457] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:20:35,468] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:20:35,482] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:20:35,494] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:20:36,459] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:36,557] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:20:36,569] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:36,671] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.153:8091',default bucket [2021-05-11 17:20:36,686] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:36,782] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:20:36,796] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:36,903] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:20:36,983] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:37,074] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:37,159] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:37,282] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:37,385] - [task:2298] INFO - 1000 items will be verified on default bucket [2021-05-11 17:20:37,905] - [task:2307] INFO - 0 items were verified [2021-05-11 17:20:38,247] - [task:2311] INFO - 1000 items were verified in 0.8611340522766113 sec.the average number of ops - 1161.0 per second [2021-05-11 17:20:38,265] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:20:38,275] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:20:38,928] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:39,023] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:39,165] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:39,275] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:39,367] - [task:956] INFO - Saw curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:20:39,379] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:39,505] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:39,601] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:39,689] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:39,788] - [task:956] INFO - Saw vb_active_curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:20:39,800] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:39,903] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:40,002] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:40,118] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:40,230] - [task:956] INFO - Saw vb_replica_curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:20:40,256] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:40,416] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:40,642] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:40,813] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:40,960] - [task:956] INFO - Saw curr_items_tot 2000 == 2000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:20:40,961] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2021-05-11 17:20:42,736] - [rebalance_helper:370] INFO - curr_items from 172.23.105.151:8091 : 249 [2021-05-11 17:20:42,737] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.151:8091 : 0 [2021-05-11 17:20:42,737] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.151:8091 : 256 [2021-05-11 17:20:42,737] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.151:8091 : 256 [2021-05-11 17:20:42,737] - [rebalance_helper:370] INFO - curr_items from 172.23.105.153:8091 : 251 [2021-05-11 17:20:42,737] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.153:8091 : 0 [2021-05-11 17:20:42,737] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.153:8091 : 256 [2021-05-11 17:20:42,737] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.153:8091 : 256 [2021-05-11 17:20:42,737] - [rebalance_helper:370] INFO - curr_items from 172.23.107.160:8091 : 251 [2021-05-11 17:20:42,738] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.107.160:8091 : 0 [2021-05-11 17:20:42,738] - [rebalance_helper:380] INFO - vb_active_num from 172.23.107.160:8091 : 256 [2021-05-11 17:20:42,738] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.107.160:8091 : 256 [2021-05-11 17:20:42,738] - [rebalance_helper:370] INFO - curr_items from 172.23.107.184:8091 : 249 [2021-05-11 17:20:42,738] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.107.184:8091 : 0 [2021-05-11 17:20:42,738] - [rebalance_helper:380] INFO - vb_active_num from 172.23.107.184:8091 : 256 [2021-05-11 17:20:42,738] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.107.184:8091 : 256 [2021-05-11 17:20:42,738] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2021-05-11 17:20:42,738] - [rebalance_helper:390] INFO - sum : 1000 and sum * (replica_factor + 1) (2) : 2000 [2021-05-11 17:20:43,530] - [rebalance_helper:393] INFO - curr_items_tot from master: 2000 [2021-05-11 17:20:43,530] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2021-05-11 17:20:43,547] - [basetestcase:3022] INFO - Servers to check bucket-seqno: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:20:43,548] - [basetestcase:2981] INFO - Record vbucket for the bucket default [2021-05-11 17:20:43,721] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:20:43,979] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:20:44,359] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:20:44,630] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:20:44,948] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:20:44,949] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.97979797979798, 'mem_free': 6573068288, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:20:44,949] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['kv'], 'cpu_utilization': 1.895854398382204, 'mem_free': 6873681920, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:20:44,949] - [basetestcase:468] INFO - 172.23.107.160:8091 => {'services': ['kv'], 'cpu_utilization': 1.986421926074931, 'mem_free': 6728269824, 'mem_total': 7934996480, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:20:44,950] - [basetestcase:468] INFO - 172.23.107.184:8091 => {'services': ['kv'], 'cpu_utilization': 3.850994210923735, 'mem_free': 6866333696, 'mem_total': 8192131072, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:20:44,950] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:20:44,953] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:20:45,098] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:20:45,497] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:20:45,862] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:20:45,865] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:20:46,017] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:20:46,493] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:20:46,865] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:20:46,868] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:20:47,036] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:20:47,406] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:20:47,813] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:20:47,816] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:20:47,956] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:20:48,391] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:20:48,737] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:20:56,722] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #3 rebalance_in_with_ops ============== [2021-05-11 17:20:56,780] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:20:58,302] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:20:58,309] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:20:58,310] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:20:58,339] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:20:58,350] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:20:58,364] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:20:58,372] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:20:58,412] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 17:20:58,415] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:20:58,451] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:20:58,457] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:21:18,487] - [rest_client:1833] INFO - rebalance progress took 20.04 seconds [2021-05-11 17:21:18,488] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:21:28,528] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091), ('ns_1@172.23.107.160', 8091), ('ns_1@172.23.107.184', 8091)] [2021-05-11 17:21:28,535] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:21:28,535] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:28,540] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:21:28,548] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:21:28,548] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:28,555] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:21:28,561] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:21:28,561] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:28,564] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:21:28,569] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:21:28,569] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:28,575] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:21:28,575] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #3 rebalance_in_with_ops ============== [2021-05-11 17:21:28,576] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 17:21:28,576] - [basetestcase:608] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_3 ok ---------------------------------------------------------------------- Ran 1 test in 260.336s OK rebalance_out_with_ops (rebalance.rebalanceout.RebalanceOutTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_4 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops,nodes_out=3,replicas=1,items=1000,services_init=kv-kv-kv-kv,services_out=kv Test Input params: {'nodes_out': '3', 'replicas': '1', 'items': '1000', 'services_init': 'kv-kv-kv-kv', 'services_out': 'kv', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 4, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_4'} [2021-05-11 17:21:28,605] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:21:28,744] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:21:29,208] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:21:29,582] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:21:29,588] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:29,602] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:21:29,616] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:21:29,616] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #4 rebalance_out_with_ops============== [2021-05-11 17:21:29,634] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:21:29,642] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:21:29,650] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:21:29,656] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:21:29,662] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:29,678] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:21:29,678] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:29,682] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:21:29,687] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:21:29,687] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:29,691] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:21:29,696] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:21:29,696] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:29,700] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:21:29,705] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:21:29,705] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:21:29,710] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:21:29,710] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:21:30,613] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:21:30,618] - [task:157] INFO - {'uptime': '1020', 'memoryTotal': 8192827392, 'memoryFree': 6618513408, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3419, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 17:21:30,626] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:21:30,634] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv']) [2021-05-11 17:21:30,634] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv [2021-05-11 17:21:30,639] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:21:30,640] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:21:30,640] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:21:30,640] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:21:30,729] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:21:30,731] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:21:30,870] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:21:31,274] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:21:31,656] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:21:31,674] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:21:31,735] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:21:31,735] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:21:31,746] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:31,756] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:31,770] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:21:31,817] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:21:31,822] - [task:157] INFO - {'uptime': '1042', 'memoryTotal': 8191283200, 'memoryFree': 6978027520, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:21:31,824] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:21:31,825] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:21:31,830] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:21:31,830] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:21:31,928] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:21:31,930] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:21:32,076] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:21:32,593] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:21:32,964] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:21:32,978] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:21:33,043] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:21:33,043] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:21:33,049] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:33,053] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:33,059] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:21:33,087] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:21:33,096] - [task:157] INFO - {'uptime': '1006', 'memoryTotal': 7934996480, 'memoryFree': 6840705024, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:21:33,104] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:21:33,104] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:21:33,115] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:21:33,115] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:21:33,209] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:21:33,211] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:21:33,317] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:21:33,619] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:21:33,925] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:21:33,952] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:21:34,138] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:21:34,138] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:21:34,151] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:34,163] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:34,179] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:21:34,224] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:21:34,227] - [task:157] INFO - {'uptime': '1008', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 7812, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:21:34,230] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:21:34,230] - [rest_client:1147] INFO - pools/default params : memoryQuota=5234 [2021-05-11 17:21:34,236] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:21:34,236] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:21:34,331] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:21:34,333] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:21:34,494] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:21:34,987] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:21:35,413] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:21:35,425] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:21:35,492] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:21:35,493] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:21:35,498] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:35,500] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:21:35,508] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:21:35,560] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:21:35,736] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:21:40,741] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:21:40,787] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:21:41,541] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 17:21:41,542] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:21:51,578] - [rest_client:1833] INFO - rebalance progress took 10.04 seconds [2021-05-11 17:21:51,578] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:22:10,337] - [task:769] INFO - adding node 172.23.107.160:8091 to cluster [2021-05-11 17:22:10,338] - [rest_client:1500] INFO - adding remote node @172.23.107.160:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:22:10,343] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/pools/default/rebalanceProgress error [Errno 111] Connection refused [2021-05-11 17:22:23,366] - [rest_client:1833] INFO - rebalance progress took 13.03 seconds [2021-05-11 17:22:23,366] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:22:43,421] - [task:769] INFO - adding node 172.23.107.184:8091 to cluster [2021-05-11 17:22:43,421] - [rest_client:1500] INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:22:53,453] - [rest_client:1833] INFO - rebalance progress took 10.03 seconds [2021-05-11 17:22:53,453] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:23:12,482] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/nodeStatuses body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password [2021-05-11 17:23:12,502] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password http://172.23.105.151:8091/pools/default with status False: ['Unexpected server error, request logged.'] [2021-05-11 17:23:12,503] - [rest_client:402] ERROR - Error while processing cluster info ['Unexpected server error, request logged.'] [2021-05-11 17:23:12,503] - [rest_client:1727] INFO - rebalance params : {'knownNodes': '', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:23:12,507] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/controller/rebalance error [Errno 111] Connection refused [2021-05-11 17:23:15,522] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/controller/rebalance body: knownNodes=&ejectedNodes=&user=Administrator&password=password headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"empty_known_nodes":1}' auth: Administrator:password [2021-05-11 17:23:15,522] - [rest_client:1734] ERROR - rebalance operation failed: b'{"empty_known_nodes":1}' [, , , , , ] Tue May 11 17:23:15 2021 [, , , , , , , , , , , ] Cluster instance shutdown with force [2021-05-11 17:23:15,527] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:23:15,529] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:23:15,531] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:23:15,533] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:23:15,694] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:23:15,713] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:23:15,718] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:23:15,754] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:23:16,091] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:23:16,165] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:23:16,238] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:23:16,262] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:23:16,483] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.160 [2021-05-11 17:23:16,484] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/cbcollect_info 172.23.107.160-20210511-1723-diag.zip [2021-05-11 17:23:16,611] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.151 [2021-05-11 17:23:16,612] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbcollect_info 172.23.105.151-20210511-1723-diag.zip [2021-05-11 17:23:16,659] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.184 [2021-05-11 17:23:16,660] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/cbcollect_info 172.23.107.184-20210511-1723-diag.zip [2021-05-11 17:23:16,720] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.153 [2021-05-11 17:23:16,721] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/cbcollect_info 172.23.105.153-20210511-1723-diag.zip [2021-05-11 17:25:59,099] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:25:59,985] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.107.160 [2021-05-11 17:26:00,390] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -f /root/172.23.107.160-20210511-1723-diag.zip [2021-05-11 17:26:00,453] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.105.153 [2021-05-11 17:26:01,164] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -f /root/172.23.105.153-20210511-1723-diag.zip [2021-05-11 17:26:01,205] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:26:16,062] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.107.184 [2021-05-11 17:26:17,207] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -f /root/172.23.107.184-20210511-1723-diag.zip [2021-05-11 17:26:17,244] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:28:03,254] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.105.151 [2021-05-11 17:28:04,217] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -f /root/172.23.105.151-20210511-1723-diag.zip [2021-05-11 17:28:04,247] - [remote_util:3475] INFO - command executed successfully with root grabbing dbinfo from 172.23.105.151 [2021-05-11 17:28:04,250] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:28:04,390] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:28:04,832] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:28:05,193] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.105.151 [2021-05-11 17:28:05,194] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.105.151-20210511-1728-couch-dbinfo.txt [2021-05-11 17:28:05,280] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:28:05,281] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file [2021-05-11 17:28:05,348] - [remote_util:1500] INFO - File 172.23.105.151-20210511-1728-couch-dbinfo.txt will be deleted [2021-05-11 17:28:05,349] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /root/*172.23.105.151-20210511-1728-couch-dbinfo.txt* [2021-05-11 17:28:05,368] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:28:05,369] - [remote_util:338] INFO - 172.23.105.151:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.105.153 [2021-05-11 17:28:35,402] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:28:35,548] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:28:36,034] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:28:36,416] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.105.153 [2021-05-11 17:28:36,417] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.105.153-20210511-1728-couch-dbinfo.txt [2021-05-11 17:28:36,509] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:28:36,509] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file [2021-05-11 17:28:36,608] - [remote_util:1500] INFO - File 172.23.105.153-20210511-1728-couch-dbinfo.txt will be deleted [2021-05-11 17:28:36,608] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /root/*172.23.105.153-20210511-1728-couch-dbinfo.txt* [2021-05-11 17:28:36,632] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:28:36,632] - [remote_util:338] INFO - 172.23.105.153:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.107.160 [2021-05-11 17:29:06,652] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:29:06,765] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:29:07,095] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:29:07,434] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.107.160 [2021-05-11 17:29:07,435] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.160-20210511-1729-couch-dbinfo.txt [2021-05-11 17:29:07,508] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:29:07,508] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file [2021-05-11 17:29:07,573] - [remote_util:1500] INFO - File 172.23.107.160-20210511-1729-couch-dbinfo.txt will be deleted [2021-05-11 17:29:07,574] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /root/*172.23.107.160-20210511-1729-couch-dbinfo.txt* [2021-05-11 17:29:07,589] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:29:07,589] - [remote_util:338] INFO - 172.23.107.160:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.107.184 [2021-05-11 17:29:37,624] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:29:37,773] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:29:38,270] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:29:38,624] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.107.184 [2021-05-11 17:29:38,625] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.184-20210511-1729-couch-dbinfo.txt [2021-05-11 17:29:38,673] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:29:38,673] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0": no header in non-empty file Failed to open "/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0": no header in non-empty file [2021-05-11 17:29:38,778] - [remote_util:1500] INFO - File 172.23.107.184-20210511-1729-couch-dbinfo.txt will be deleted [2021-05-11 17:29:38,778] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /root/*172.23.107.184-20210511-1729-couch-dbinfo.txt* [2021-05-11 17:29:38,801] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:29:38,802] - [remote_util:338] INFO - 172.23.107.184:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_4 Traceback (most recent call last): File "lib/tasks/task.py", line 756, in execute self.start_rebalance(task_manager) File "lib/tasks/task.py", line 815, in start_rebalance self.rest.rebalance(otpNodes=[node.id for node in nodes], ejectedNodes=ejectedNodes) File "lib/membase/api/rest_client.py", line 1737, in rebalance parameters=params) membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password Traceback (most recent call last): File "pytests/basetestcase.py", line 369, in setUp services=self.services) File "lib/couchbase_helper/cluster.py", line 415, in rebalance return _task.result(timeout) File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception File "lib/tasks/task.py", line 756, in execute self.start_rebalance(task_manager) File "lib/tasks/task.py", line 815, in start_rebalance self.rest.rebalance(otpNodes=[node.id for node in nodes], ejectedNodes=ejectedNodes) File "lib/membase/api/rest_client.py", line 1737, in rebalance parameters=params) membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password Traceback (most recent call last): File "pytests/basetestcase.py", line 369, in setUp services=self.services) File "lib/couchbase_helper/cluster.py", line 415, in rebalance return _task.result(timeout) File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception File "lib/tasks/task.py", line 756, in execute self.start_rebalance(task_manager) File "lib/tasks/task.py", line 815, in start_rebalance self.rest.rebalance(otpNodes=[node.id for node in nodes], ejectedNodes=ejectedNodes) File "lib/membase/api/rest_client.py", line 1737, in rebalance parameters=params) membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/basetestcase.py", line 396, in setUp self.fail(e) File "/usr/local/lib/python3.7/unittest/case.py", line 693, in fail raise self.failureException(msg) AssertionError: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password FAIL ====================================================================== FAIL: rebalance_out_with_ops (rebalance.rebalanceout.RebalanceOutTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/basetestcase.py", line 369, in setUp services=self.services) File "lib/couchbase_helper/cluster.py", line 415, in rebalance return _task.result(timeout) File "lib/tasks/future.py", line 160, in result return self.__get_result() membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/basetestcase.py", line 396, in setUp self.fail(e) File "/usr/local/lib/python3.7/unittest/case.py", line 693, in fail raise self.failureException(msg) AssertionError: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password During handling of the above exception, another exception occurred: Traceback (most recent call last): File "pytests/rebalance/rebalanceout.py", line 17, in setUp super(RebalanceOutTests, self).setUp() File "pytests/rebalance/rebalance_base.py", line 11, in setUp super(RebalanceBaseTest, self).setUp() File "pytests/basetestcase.py", line 455, in setUp self.fail(e) AssertionError: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password ---------------------------------------------------------------------- Ran 1 test in 106.922s FAILED (failures=1) do_test (swaprebalance.SwapRebalanceBasicTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_5 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=1,num-buckets=1,num-swap=1,items=1000 Test Input params: {'replica': '1', 'num-buckets': '1', 'num-swap': '1', 'items': '1000', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 5, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_5'} [2021-05-11 17:30:08,832] - [swaprebalance:56] INFO - ============== SwapRebalanceBase setup was started for test #5 do_test============== [2021-05-11 17:30:08,833] - [swaprebalance:111] INFO - ============== SwapRebalanceBase cleanup was started for test #5 do_test ============== [2021-05-11 17:30:08,833] - [swaprebalance:112] INFO - Stopping load in Teardown [2021-05-11 17:30:08,876] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:30:08,885] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:30:08,894] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:30:08,903] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:30:08,909] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:08,924] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 17:30:08,927] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:30:08,958] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:30:08,962] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:30:28,992] - [rest_client:1833] INFO - rebalance progress took 20.03 seconds [2021-05-11 17:30:28,992] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:30:39,082] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091), ('ns_1@172.23.107.160', 8091), ('ns_1@172.23.107.184', 8091)] [2021-05-11 17:30:39,096] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,147] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,196] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,245] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:30:39,245] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,251] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:30:39,265] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:30:39,265] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,278] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:30:39,285] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:30:39,285] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,289] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:30:39,302] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:30:39,302] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:30:39,315] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:30:39,316] - [swaprebalance:128] INFO - ============== SwapRebalanceBase cleanup was finished for test #5 do_test ============== [2021-05-11 17:30:39,316] - [swaprebalance:64] INFO - picking server : ip:172.23.105.151 port:8091 ssh_username:root as the master [2021-05-11 17:30:39,322] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:30:39,323] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:30:39,436] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:30:39,436] - [rest_client:1147] INFO - pools/default params : memoryQuota=4166 [2021-05-11 17:30:39,449] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:30:39,654] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:30:40,178] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:30:40,623] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:30:40,639] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:30:40,711] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:30:40,712] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:30:40,712] - [swaprebalance:143] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:30:40,914] - [swaprebalance:81] INFO - ============== SwapRebalanceBase setup was finished for test #5 do_test ============== [2021-05-11 17:30:40,929] - [swaprebalance:287] INFO - CREATE BUCKET PHASE [2021-05-11 17:30:40,945] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:30:40,965] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=2777&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:30:40,998] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 17:30:40,999] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:30:41,808] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:30:42,073] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:30:43,057] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['default'] [2021-05-11 17:30:43,057] - [rest_client:141] INFO - node 172.23.105.151 found bucket default [2021-05-11 17:30:43,058] - [swaprebalance:291] INFO - INITIAL REBALANCE PHASE [2021-05-11 17:30:43,089] - [rebalance_helper:460] INFO - current nodes : ['172.23.105.151:8091'] [2021-05-11 17:30:43,089] - [rebalance_helper:475] INFO - choosing 172.23.105.153:8091 [2021-05-11 17:30:43,089] - [rebalance_helper:475] INFO - choosing 172.23.107.160:8091 [2021-05-11 17:30:43,089] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:30:53,104] - [rest_client:1833] INFO - rebalance progress took 10.01 seconds [2021-05-11 17:30:53,104] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:31:12,039] - [rest_client:1500] INFO - adding remote node @172.23.107.160:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:31:12,040] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/pools/default/rebalanceProgress error [Errno 111] Connection refused [2021-05-11 17:31:25,062] - [rest_client:1833] INFO - rebalance progress took 13.02 seconds [2021-05-11 17:31:25,062] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:31:43,889] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/nodeStatuses body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password [2021-05-11 17:31:43,890] - [rest_client:1727] INFO - rebalance params : {'knownNodes': '', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:31:43,900] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/controller/rebalance error [Errno 111] Connection refused [2021-05-11 17:31:46,908] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/controller/rebalance body: knownNodes=&ejectedNodes=&user=Administrator&password=password headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"empty_known_nodes":1}' auth: Administrator:password [2021-05-11 17:31:46,908] - [rest_client:1734] ERROR - rebalance operation failed: b'{"empty_known_nodes":1}' [2021-05-11 17:31:46,909] - [swaprebalance:111] INFO - ============== SwapRebalanceBase cleanup was started for test #5 do_test ============== [2021-05-11 17:31:46,909] - [swaprebalance:112] INFO - Stopping load in Teardown [2021-05-11 17:31:46,970] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/pools/default/buckets?basic_stats=true error [Errno 111] Connection refused [2021-05-11 17:31:50,016] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:31:59,756] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:31:59,760] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:31:59,761] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:31:59,772] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:31:59,788] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:31:59,798] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:31:59,808] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:31:59,833] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 17:31:59,837] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153,ns_1@172.23.107.160', 'ejectedNodes': 'ns_1@172.23.105.153,ns_1@172.23.107.160', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:31:59,875] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:31:59,882] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:32:09,897] - [rest_client:1894] INFO - rebalance percentage : 80.00 % [2021-05-11 17:32:19,912] - [rest_client:1894] INFO - rebalance percentage : 83.00 % [2021-05-11 17:32:39,947] - [rest_client:1833] INFO - rebalance progress took 40.07 seconds [2021-05-11 17:32:39,947] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:32:49,992] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091), ('ns_1@172.23.107.160', 8091)] [2021-05-11 17:32:49,999] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,024] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,068] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,093] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:32:50,093] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,104] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:32:50,108] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:32:50,108] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,113] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:32:50,124] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:32:50,124] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,134] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:32:50,139] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:32:50,139] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:32:50,143] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:32:50,143] - [swaprebalance:128] INFO - ============== SwapRebalanceBase cleanup was finished for test #5 do_test ============== [2021-05-11 17:32:50,202] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:32:50,204] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:32:50,206] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:32:50,209] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:32:50,315] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:32:50,363] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:32:50,370] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:32:50,374] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:32:50,628] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:32:50,839] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:32:50,877] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:32:50,920] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:32:50,939] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.160 [2021-05-11 17:32:50,940] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/cbcollect_info 172.23.107.160-20210511-1732-diag.zip [2021-05-11 17:32:51,303] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.107.184 [2021-05-11 17:32:51,304] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/cbcollect_info 172.23.107.184-20210511-1732-diag.zip [2021-05-11 17:32:51,374] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.153 [2021-05-11 17:32:51,375] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/cbcollect_info 172.23.105.153-20210511-1732-diag.zip [2021-05-11 17:32:51,485] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.105.151 [2021-05-11 17:32:51,486] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbcollect_info 172.23.105.151-20210511-1732-diag.zip [2021-05-11 17:35:26,665] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.107.160 [2021-05-11 17:35:27,533] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -f /root/172.23.107.160-20210511-1732-diag.zip [2021-05-11 17:35:27,558] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:35:28,101] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.105.153 [2021-05-11 17:35:29,119] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -f /root/172.23.105.153-20210511-1732-diag.zip [2021-05-11 17:35:29,152] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:35:34,421] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.107.184 [2021-05-11 17:35:35,282] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -f /root/172.23.107.184-20210511-1732-diag.zip [2021-05-11 17:35:35,308] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:37:34,536] - [remote_util:3475] INFO - command executed successfully with root Downloading zipped logs from 172.23.105.151 [2021-05-11 17:37:37,057] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -f /root/172.23.105.151-20210511-1732-diag.zip [2021-05-11 17:37:37,091] - [remote_util:3475] INFO - command executed successfully with root grabbing dbinfo from 172.23.105.151 [2021-05-11 17:37:37,094] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:37:37,225] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:37:37,631] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:37:37,987] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.105.151 [2021-05-11 17:37:37,988] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.105.151-20210511-1737-couch-dbinfo.txt [2021-05-11 17:37:38,075] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:37:38,075] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file Failed to open "/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0": malformed data in file [2021-05-11 17:37:38,146] - [remote_util:1500] INFO - File 172.23.105.151-20210511-1737-couch-dbinfo.txt will be deleted [2021-05-11 17:37:38,146] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /root/*172.23.105.151-20210511-1737-couch-dbinfo.txt* [2021-05-11 17:37:38,168] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:37:38,168] - [remote_util:338] INFO - 172.23.105.151:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.105.153 [2021-05-11 17:38:08,201] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:38:08,341] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:38:08,787] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:38:09,160] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.105.153 [2021-05-11 17:38:09,161] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.105.153-20210511-1738-couch-dbinfo.txt [2021-05-11 17:38:09,249] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:38:09,250] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file [2021-05-11 17:38:09,351] - [remote_util:1500] INFO - File 172.23.105.153-20210511-1738-couch-dbinfo.txt will be deleted [2021-05-11 17:38:09,351] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /root/*172.23.105.153-20210511-1738-couch-dbinfo.txt* [2021-05-11 17:38:09,375] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:38:09,375] - [remote_util:338] INFO - 172.23.105.153:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.107.160 [2021-05-11 17:38:39,389] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:38:39,497] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:38:39,809] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:38:40,147] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.107.160 [2021-05-11 17:38:40,147] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.160-20210511-1738-couch-dbinfo.txt [2021-05-11 17:38:40,217] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:38:40,217] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file [2021-05-11 17:38:40,278] - [remote_util:1500] INFO - File 172.23.107.160-20210511-1738-couch-dbinfo.txt will be deleted [2021-05-11 17:38:40,278] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /root/*172.23.107.160-20210511-1738-couch-dbinfo.txt* [2021-05-11 17:38:40,292] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:38:40,292] - [remote_util:338] INFO - 172.23.107.160:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... grabbing dbinfo from 172.23.107.184 [2021-05-11 17:39:10,326] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:39:10,531] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:39:10,994] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:39:11,420] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting dbinfo from 172.23.107.184 [2021-05-11 17:39:11,421] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.184-20210511-1739-couch-dbinfo.txt [2021-05-11 17:39:11,480] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:39:11,480] - [remote_util:3260] INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0": no header in non-empty file Failed to open "/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0": no header in non-empty file [2021-05-11 17:39:11,653] - [remote_util:1500] INFO - File 172.23.107.184-20210511-1739-couch-dbinfo.txt will be deleted [2021-05-11 17:39:11,653] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /root/*172.23.107.184-20210511-1739-couch-dbinfo.txt* [2021-05-11 17:39:11,683] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:39:11,684] - [remote_util:338] INFO - 172.23.107.184:sleep for 30 secs. ** Network or sever may be busy. ** Wait 30 seconds before executing next instrucion ... summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_5 ERROR ====================================================================== ERROR: do_test (swaprebalance.SwapRebalanceBasicTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/swaprebalance.py", line 660, in do_test SwapRebalanceBase._common_test_body_swap_rebalance(self, do_stop_start=False) File "pytests/swaprebalance.py", line 292, in _common_test_body_swap_rebalance status, servers_rebalanced = RebalanceHelper.rebalance_in(intial_severs, len(intial_severs) - 1) File "lib/membase/helper/rebalance_helper.py", line 486, in rebalance_in started = rest.rebalance(otpNodes, []) File "lib/membase/api/rest_client.py", line 1737, in rebalance parameters=params) membase.api.exception.InvalidArgumentException: controller/rebalance with error message b'{"empty_known_nodes":1}' failed when invoked with parameters: knownNodes=&ejectedNodes=&user=Administrator&password=password ---------------------------------------------------------------------- Ran 1 test in 161.375s FAILED (errors=1) test_failover_normal (failover.failovertests.FailoverTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_6 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,failover.failovertests.FailoverTests.test_failover_normal,replica=1,load_ratio=1,num_failed_nodes=1,withMutationOps=True,services_init=kv-kv-kv-kv,services_out=kv Test Input params: {'replica': '1', 'load_ratio': '1', 'num_failed_nodes': '1', 'withMutationOps': 'True', 'services_init': 'kv-kv-kv-kv', 'services_out': 'kv', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 6, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_6'} [2021-05-11 17:39:41,732] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:39:41,871] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:39:42,294] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:39:42,646] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:39:42,653] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:39:42,667] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:39:42,676] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:39:42,676] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #6 test_failover_normal============== [2021-05-11 17:39:42,693] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:39:42,702] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:39:42,709] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:39:42,716] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:39:42,721] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:39:42,738] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:39:42,738] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:39:42,742] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:39:42,747] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:39:42,747] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:39:42,750] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:39:42,754] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:39:42,755] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:39:42,758] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:39:42,763] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:39:42,763] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:39:42,767] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:39:42,768] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:39:43,739] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:39:43,744] - [task:157] INFO - {'uptime': '2113', 'memoryTotal': 8192827392, 'memoryFree': 6742732800, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4166, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 17:39:43,751] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:39:43,758] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv']) [2021-05-11 17:39:43,758] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv [2021-05-11 17:39:43,762] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:39:43,763] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:39:43,763] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:39:43,763] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:39:43,853] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:39:43,856] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:39:43,995] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:39:44,374] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:39:44,727] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:39:44,738] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:39:44,800] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:39:44,800] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:39:44,803] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:44,805] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:44,811] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:39:44,838] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:39:44,842] - [task:157] INFO - {'uptime': '2138', 'memoryTotal': 8191283200, 'memoryFree': 6941474816, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:39:44,844] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:39:44,844] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:39:44,849] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:39:44,850] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:39:44,950] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:39:44,952] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:39:45,086] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:39:45,529] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:39:45,906] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:39:45,917] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:39:45,983] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:39:45,983] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:39:45,986] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:45,988] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:45,993] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:39:46,027] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:39:46,036] - [task:157] INFO - {'uptime': '2102', 'memoryTotal': 7934996480, 'memoryFree': 6823145472, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:39:46,045] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:39:46,046] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:39:46,056] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:39:46,056] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:39:46,162] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:39:46,164] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:39:46,329] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:39:46,698] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:39:47,084] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:39:47,110] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:39:47,307] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:39:47,308] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:39:47,319] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:47,328] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:47,341] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:39:47,385] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:39:47,396] - [task:157] INFO - {'uptime': '2105', 'memoryTotal': 8192131072, 'memoryFree': 6962216960, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:39:47,405] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:39:47,405] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:39:47,417] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:39:47,417] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:39:47,519] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:39:47,521] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:39:47,697] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:39:48,152] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:39:48,565] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:39:48,594] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:39:48,654] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:39:48,654] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:39:48,667] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:48,678] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:39:48,692] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:39:48,757] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:39:48,799] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 17:39:48,800] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 17:39:48,921] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:39:53,926] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:39:53,942] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:39:54,737] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 17:39:54,738] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:40:04,746] - [rest_client:1833] INFO - rebalance progress took 10.01 seconds [2021-05-11 17:40:04,746] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:40:23,448] - [task:769] INFO - adding node 172.23.107.160:8091 to cluster [2021-05-11 17:40:23,449] - [rest_client:1500] INFO - adding remote node @172.23.107.160:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:40:33,469] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 17:40:33,469] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:40:52,163] - [task:769] INFO - adding node 172.23.107.184:8091 to cluster [2021-05-11 17:40:52,164] - [rest_client:1500] INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:41:02,190] - [rest_client:1833] INFO - rebalance progress took 10.03 seconds [2021-05-11 17:41:02,191] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:41:21,540] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:41:21,590] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:41:31,612] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 17:41:31,629] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.038012266159058 sec [2021-05-11 17:41:31,631] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:41:31,793] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:41:32,269] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:41:32,722] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:41:32,752] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:41:32,810] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:41:32,811] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:41:32,811] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:41:33,704] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=4187&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:41:33,745] - [rest_client:2843] INFO - 0.04 seconds to create bucket default [2021-05-11 17:41:33,746] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:41:34,705] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:35,000] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:35,268] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:35,589] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:37,020] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:37,252] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:37,460] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:37,628] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:37,833] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:41:37,834] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:41:38,093] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:38,255] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:38,435] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:38,604] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:38,873] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:39,036] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:39,200] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:39,385] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:39,541] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:41:39,542] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:41:39,804] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:39,968] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:40,120] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:40,313] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:40,588] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:40,803] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:41,008] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:41,227] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:41,422] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:41:41,423] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:41:41,771] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:41,990] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:42,267] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:42,497] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:42,836] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:43,079] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:43,330] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:43,585] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:43,773] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:41:43,774] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:41:44,016] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:44,202] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:44,442] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:44,662] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:44,948] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:45,120] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:45,307] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:45,478] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:45,634] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:41:45,634] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:41:45,910] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:46,083] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:46,260] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:46,447] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:46,616] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:41:46,772] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:41:46,885] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:41:47,056] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:41:47,125] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:41:47,128] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #6 test_failover_normal ============== [2021-05-11 17:41:47,155] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:41:47,316] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:41:47,764] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:41:48,205] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:41:48,208] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:41:48,380] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:41:48,878] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:41:49,314] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:41:49,318] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:41:49,465] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:41:49,782] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:41:50,097] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:41:50,099] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:41:50,259] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:41:50,739] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:41:51,171] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:42:06,868] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:42:06,868] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.216733870967742, 'mem_free': 6629756928, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:42:06,868] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['kv'], 'cpu_utilization': 15.78005115089514, 'mem_free': 6875078656, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:42:06,868] - [basetestcase:468] INFO - 172.23.107.160:8091 => {'services': ['kv'], 'cpu_utilization': 7.267221801665405, 'mem_free': 6745268224, 'mem_total': 7934996480, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:42:06,868] - [basetestcase:468] INFO - 172.23.107.184:8091 => {'services': ['kv'], 'cpu_utilization': 1.987921489682939, 'mem_free': 6894796800, 'mem_total': 8192131072, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:42:06,868] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:42:06,869] - [failoverbasetests:85] INFO - ============== FailoverBaseTest setup was finished for test #6 test_failover_normal ============== [2021-05-11 17:42:06,869] - [failovertests:49] INFO - Picking node 172.23.105.151 as reference node for test case [2021-05-11 17:42:06,869] - [failovertests:319] INFO - num_replicas : 1 [2021-05-11 17:42:06,869] - [failovertests:320] INFO - recoveryType : ['delta'] [2021-05-11 17:42:06,869] - [failovertests:321] INFO - failover_reason : normal [2021-05-11 17:42:06,869] - [failovertests:322] INFO - num_failed_nodes : 1 [2021-05-11 17:42:06,869] - [failovertests:323] INFO - picking server : ip:172.23.105.151 port:8091 ssh_username:root as the master [2021-05-11 17:42:06,922] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:42:06,922] - [basetestcase:662] INFO - sleep for 120 secs. ... [2021-05-11 17:44:07,144] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:07,271] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:07,378] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:07,485] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:08,092] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:08,094] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:08,095] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:08,097] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:08,205] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:08,206] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:08,207] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:08,216] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:08,359] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:08,359] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:08,361] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:08,369] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:08,465] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:08,466] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:08,468] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:08,469] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:08,809] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:44:08,824] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:44:09,001] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:09,093] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:44:09,105] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:09,540] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:44:09,556] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:09,655] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.153:8091',default bucket [2021-05-11 17:44:09,672] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:10,037] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.153:8091',default bucket [2021-05-11 17:44:10,052] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:10,182] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:44:10,194] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:10,578] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:44:10,590] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:10,680] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:44:10,691] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:11,099] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:44:11,119] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:44:11,133] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:44:12,114] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:12,220] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:12,340] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:12,479] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:12,596] - [task:956] INFO - Saw curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:44:12,613] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:12,723] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:12,848] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:12,954] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:13,057] - [task:956] INFO - Saw vb_active_curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:44:13,071] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:13,224] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:13,328] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:13,420] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:13,536] - [task:956] INFO - Saw vb_replica_curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:44:13,551] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:13,655] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:13,767] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:13,895] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:14,004] - [task:956] INFO - Saw curr_items_tot 2000 == 2000 expected on '172.23.105.151:8091''172.23.105.153:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:44:14,025] - [basetestcase:3022] INFO - Servers to check bucket-seqno: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:44:14,026] - [basetestcase:2981] INFO - Record vbucket for the bucket default [2021-05-11 17:44:14,183] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:14,439] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:14,686] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:14,949] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:15,211] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:15,491] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:15,778] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:16,057] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:16,363] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:16,513] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:16,621] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:16,728] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:16,861] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:17,047] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:44:17,238] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:17,446] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:18,066] - [task:4057] INFO - Failing over 172.23.105.153:8091 with graceful=True [2021-05-11 17:44:18,122] - [rest_client:1672] INFO - fail_over node ns_1@172.23.105.153 successful [2021-05-11 17:44:18,122] - [task:4037] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2021-05-11 17:44:18,127] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:44:28,142] - [rest_client:1894] INFO - rebalance percentage : 100.00 % [2021-05-11 17:44:38,158] - [rest_client:1833] INFO - rebalance progress took 20.04 seconds [2021-05-11 17:44:38,159] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:44:48,191] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:48,191] - [basetestcase:3022] INFO - Servers to check bucket-seqno: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:44:48,191] - [basetestcase:2981] INFO - Record vbucket for the bucket default [2021-05-11 17:44:48,286] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:48,370] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:48,577] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:48,638] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:48,812] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:48,879] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,073] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,073] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:44:49,087] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,087] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:44:49,173] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:49,269] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,283] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:44:49,297] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:44:49,358] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,539] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:44:49,552] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:49,613] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,627] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:44:49,638] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:44:49,738] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:49,939] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:44:49,951] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:50,024] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:50,041] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:44:50,056] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:44:50,129] - [rest_client:2449] INFO - Node 172.23.105.153 not part of cluster inactiveFailed [2021-05-11 17:44:50,320] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:44:50,321] - [basetestcase:662] INFO - sleep for 5 secs. after failover before invoking rebalance... ... [2021-05-11 17:44:55,326] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153,ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:44:55,366] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:44:55,370] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:45:15,398] - [rest_client:1833] INFO - rebalance progress took 20.03 seconds [2021-05-11 17:45:15,398] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:45:25,408] - [failovertests:197] INFO - Begin VERIFICATION for Rebalance after Failover Only [2021-05-11 17:45:25,425] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:45:25,436] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:45:25,447] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:45:25,458] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:45:26,371] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:26,489] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:45:26,501] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:26,978] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:45:26,992] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:27,076] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:45:27,088] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:27,561] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 17:45:27,574] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:27,656] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:45:27,668] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:28,106] - [task:956] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 17:45:28,191] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:28,276] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:28,363] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:28,494] - [task:2298] INFO - 1000 items will be verified on default bucket [2021-05-11 17:45:29,109] - [task:2307] INFO - 0 items were verified [2021-05-11 17:45:29,503] - [task:2311] INFO - 1000 items were verified in 1.008758783340454 sec.the average number of ops - 991.0 per second [2021-05-11 17:45:29,525] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:45:29,535] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:45:30,130] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:30,241] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:30,358] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:30,470] - [task:956] INFO - Saw curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:45:30,482] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:30,577] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:30,670] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:30,771] - [task:956] INFO - Saw vb_active_curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:45:30,783] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:30,876] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:30,976] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:31,082] - [task:956] INFO - Saw vb_replica_curr_items 1000 == 1000 expected on '172.23.105.151:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:45:31,094] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:31,186] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:31,289] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:31,393] - [task:956] INFO - Saw curr_items_tot 2000 == 2000 expected on '172.23.105.151:8091''172.23.107.160:8091''172.23.107.184:8091',default bucket [2021-05-11 17:45:31,393] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2021-05-11 17:45:32,525] - [rebalance_helper:370] INFO - curr_items from 172.23.105.151:8091 : 333 [2021-05-11 17:45:32,525] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.151:8091 : 0 [2021-05-11 17:45:32,525] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.151:8091 : 342 [2021-05-11 17:45:32,526] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.151:8091 : 342 [2021-05-11 17:45:32,526] - [rebalance_helper:370] INFO - curr_items from 172.23.107.160:8091 : 331 [2021-05-11 17:45:32,526] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.107.160:8091 : 0 [2021-05-11 17:45:32,526] - [rebalance_helper:380] INFO - vb_active_num from 172.23.107.160:8091 : 341 [2021-05-11 17:45:32,526] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.107.160:8091 : 341 [2021-05-11 17:45:32,526] - [rebalance_helper:370] INFO - curr_items from 172.23.107.184:8091 : 336 [2021-05-11 17:45:32,527] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.107.184:8091 : 0 [2021-05-11 17:45:32,527] - [rebalance_helper:380] INFO - vb_active_num from 172.23.107.184:8091 : 341 [2021-05-11 17:45:32,527] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.107.184:8091 : 341 [2021-05-11 17:45:32,527] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2021-05-11 17:45:32,527] - [rebalance_helper:390] INFO - sum : 1000 and sum * (replica_factor + 1) (2) : 2000 [2021-05-11 17:45:33,258] - [rebalance_helper:393] INFO - curr_items_tot from master: 2000 [2021-05-11 17:45:33,258] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2021-05-11 17:45:33,259] - [basetestcase:2291] INFO - Begin Verification for failovers logs comparison [2021-05-11 17:45:33,273] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:33,392] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:33,536] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:33,653] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:33,858] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:34,078] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:34,343] - [basetestcase:2298] INFO - End Verification for Failovers logs comparison [2021-05-11 17:45:34,343] - [basetestcase:1843] INFO - Begin Verification for vbucket sequence numbers comparison [2021-05-11 17:45:34,354] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:34,672] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:35,051] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:35,438] - [basetestcase:1868] INFO - End Verification for vbucket sequence numbers comparison [2021-05-11 17:45:35,474] - [basetestcase:2046] INFO - Begin Verification for vb_distribution_analysis [2021-05-11 17:45:35,488] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:45:35,498] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:45:35,507] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise', '7.0.0-5148-enterprise', '7.0.0-5148-enterprise'] [2021-05-11 17:45:35,516] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:45:35,527] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:35,616] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:35,734] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:35,824] - [basetestcase:2054] INFO - Begin Verification for Bucket default [2021-05-11 17:45:35,879] - [basetestcase:2084] INFO - End Verification for vb_distribution_analysis [2021-05-11 17:45:35,880] - [failovertests:219] INFO - End VERIFICATION for Rebalance after Failover Only [2021-05-11 17:45:35,891] - [basetestcase:3022] INFO - Servers to check bucket-seqno: [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 17:45:35,891] - [basetestcase:2981] INFO - Record vbucket for the bucket default [2021-05-11 17:45:35,991] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:45:36,335] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 17:45:36,675] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 17:45:37,052] - [failoverbasetests:98] INFO - ============== tearDown was started for test #6 test_failover_normal ============== [2021-05-11 17:45:37,054] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:45:37,192] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:45:37,597] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:45:37,951] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:45:37,951] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:45:38,048] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 17038 [2021-05-11 17:45:38,049] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /sbin/iptables -F [2021-05-11 17:45:38,111] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:38,111] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /sbin/iptables -t nat -F [2021-05-11 17:45:38,170] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:38,171] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:45:38,299] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:45:38,300] - [remote_util:4040] INFO - *** unpause memcached process *** [2021-05-11 17:45:38,300] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: killall -SIGCONT memcached [2021-05-11 17:45:38,655] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:38,655] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: killall -SIGCONT beam [2021-05-11 17:45:38,687] - [remote_util:3473] INFO - command executed with root but got an error ['beam: no process found'] ... [2021-05-11 17:45:38,687] - [remote_util:3222] ERROR - beam: no process found [2021-05-11 17:45:38,689] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:45:38,833] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:45:39,301] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:45:39,683] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:45:39,684] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:45:39,805] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 13992 [2021-05-11 17:45:39,806] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /sbin/iptables -F [2021-05-11 17:45:39,873] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:39,873] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: /sbin/iptables -t nat -F [2021-05-11 17:45:39,937] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:39,938] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:45:40,072] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:45:40,072] - [remote_util:4040] INFO - *** unpause memcached process *** [2021-05-11 17:45:40,072] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: killall -SIGCONT memcached [2021-05-11 17:45:40,475] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:40,475] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: killall -SIGCONT beam [2021-05-11 17:45:40,508] - [remote_util:3473] INFO - command executed with root but got an error ['beam: no process found'] ... [2021-05-11 17:45:40,508] - [remote_util:3222] ERROR - beam: no process found [2021-05-11 17:45:40,511] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:45:40,612] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:45:40,910] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:45:41,214] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:45:41,215] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:45:41,316] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 3927 [2021-05-11 17:45:41,316] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /sbin/iptables -F [2021-05-11 17:45:41,377] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:41,377] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /sbin/iptables -t nat -F [2021-05-11 17:45:41,432] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:41,432] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:45:41,529] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:45:41,530] - [remote_util:4040] INFO - *** unpause memcached process *** [2021-05-11 17:45:41,530] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: killall -SIGCONT memcached [2021-05-11 17:45:41,811] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:41,811] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: killall -SIGCONT beam [2021-05-11 17:45:41,835] - [remote_util:3473] INFO - command executed with root but got an error ['beam: no process found'] ... [2021-05-11 17:45:41,835] - [remote_util:3222] ERROR - beam: no process found [2021-05-11 17:45:41,837] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:45:41,979] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:45:42,418] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:45:42,760] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:45:42,760] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:45:42,860] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 4116 [2021-05-11 17:45:42,861] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /sbin/iptables -F [2021-05-11 17:45:42,923] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:42,923] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: /sbin/iptables -t nat -F [2021-05-11 17:45:42,982] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:42,983] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:45:43,116] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:45:43,116] - [remote_util:4040] INFO - *** unpause memcached process *** [2021-05-11 17:45:43,116] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: killall -SIGCONT memcached [2021-05-11 17:45:43,487] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:45:43,487] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: killall -SIGCONT beam [2021-05-11 17:45:43,516] - [remote_util:3473] INFO - command executed with root but got an error ['beam: no process found'] ... [2021-05-11 17:45:43,516] - [remote_util:3222] ERROR - beam: no process found [2021-05-11 17:45:53,574] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:45:54,512] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:45:54,517] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:45:54,517] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:45:54,526] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:45:54,536] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:45:54,547] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:45:54,554] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:45:54,571] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 17:45:54,573] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:45:54,623] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:45:54,626] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:46:14,654] - [rest_client:1833] INFO - rebalance progress took 20.03 seconds [2021-05-11 17:46:14,654] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:46:24,682] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.107.160', 8091), ('ns_1@172.23.107.184', 8091)] [2021-05-11 17:46:24,687] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:24,702] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:24,716] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:24,725] - [failoverbasetests:109] INFO - ============== tearDown was finished for test #6 test_failover_normal ============== [2021-05-11 17:46:24,736] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:46:24,736] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.054746358613762, 'mem_free': 6575325184, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:46:24,736] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:46:24,738] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:46:24,867] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:46:25,256] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:25,609] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:25,612] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:46:25,748] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:46:26,221] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:26,591] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:26,594] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:46:26,693] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:46:26,999] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:27,319] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:27,321] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:46:27,454] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:46:27,860] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:28,205] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:45,149] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #6 test_failover_normal ============== [2021-05-11 17:46:45,171] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:46:45,179] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:46:45,186] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:46:45,192] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:46:45,198] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:45,215] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:46:45,215] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:45,220] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:46:45,224] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:46:45,224] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:45,227] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:46:45,232] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:46:45,232] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:45,236] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:46:45,240] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:46:45,240] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:45,244] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:46:45,244] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #6 test_failover_normal ============== [2021-05-11 17:46:45,244] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 17:46:45,245] - [basetestcase:608] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_6 ok ---------------------------------------------------------------------- Ran 1 test in 423.523s OK test_check_default_zone_create_by_default (rackzone.rackzonetests.RackzoneTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_7 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t rackzone.rackzonetests.RackzoneTests.test_check_default_zone_create_by_default,items=1000 Test Input params: {'items': '1000', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 7, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_7'} [2021-05-11 17:46:45,270] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:46:45,399] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:46:45,800] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:46,153] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:46,160] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:46,174] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:46:46,184] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:46:46,184] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #7 test_check_default_zone_create_by_default============== [2021-05-11 17:46:46,204] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:46:46,211] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:46:46,217] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:46:46,224] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:46:46,228] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:46,245] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:46:46,245] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:46,249] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:46:46,253] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:46:46,253] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:46,257] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:46:46,262] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:46:46,262] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:46,266] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:46:46,270] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:46:46,270] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:46:46,274] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:46:46,274] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:46:47,277] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:46:47,282] - [task:157] INFO - {'uptime': '2537', 'memoryTotal': 8192827392, 'memoryFree': 6606475264, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 17:46:47,288] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 17:46:47,288] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:46:47,288] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:46:47,297] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 17:46:47,297] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:46:47,297] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:46:47,308] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 17:46:47,314] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 17:46:47,315] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 17:46:47,318] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:46:47,319] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:46:47,319] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:46:47,319] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:46:47,400] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:46:47,402] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:46:47,528] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:46:47,888] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:48,233] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:48,247] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:46:48,308] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:46:48,308] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:46:48,312] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:48,314] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:48,320] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:46:48,350] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:46:48,355] - [task:157] INFO - {'uptime': '2560', 'memoryTotal': 8191283200, 'memoryFree': 6960259072, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:46:48,358] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:46:48,358] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:46:48,363] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:46:48,363] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:46:48,558] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:46:48,560] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:46:48,695] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:46:49,134] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:49,507] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:49,521] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:46:49,586] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:46:49,586] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:46:49,589] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:49,591] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:49,597] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:46:49,615] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:46:49,619] - [task:157] INFO - {'uptime': '2526', 'memoryTotal': 7934996480, 'memoryFree': 6825648128, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:46:49,621] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:46:49,621] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:46:49,627] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:46:49,627] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:46:49,719] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:46:49,721] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:46:49,819] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:46:50,127] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:50,435] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:50,450] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:46:50,647] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:46:50,647] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:46:50,651] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:50,654] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:50,662] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:46:50,680] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:46:50,684] - [task:157] INFO - {'uptime': '2528', 'memoryTotal': 8192131072, 'memoryFree': 6982746112, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:46:50,686] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:46:50,687] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:46:50,691] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:46:50,691] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:46:50,779] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:46:50,781] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:46:50,914] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:46:51,343] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:51,690] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:51,701] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:46:51,761] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:46:51,762] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:46:51,766] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:51,769] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:46:51,775] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:46:51,798] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:46:51,915] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:46:56,920] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:46:56,936] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:46:56,938] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:46:57,072] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:46:57,450] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:46:57,810] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:46:57,822] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:46:57,885] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:46:57,885] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:46:57,885] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:46:58,821] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3419&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:46:58,853] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 17:46:58,853] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:46:59,603] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:46:59,921] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:00,049] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:47:00,050] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:47:00,280] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:00,482] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:00,627] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:47:00,628] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:47:00,813] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:01,045] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:01,152] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:47:01,152] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:47:01,366] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:01,556] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:01,714] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:47:01,714] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:47:01,906] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:02,120] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:02,247] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:47:02,247] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:47:02,457] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:02,659] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:02,795] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:47:02,798] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #7 test_check_default_zone_create_by_default ============== [2021-05-11 17:47:02,807] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:47:02,929] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:47:03,310] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:03,657] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:03,660] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:47:03,792] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:47:04,241] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:04,606] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:04,609] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:47:04,712] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:47:05,054] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:05,363] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:05,367] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:47:05,508] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:47:05,925] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:06,268] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:22,584] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:47:22,584] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 21.69405815423514, 'mem_free': 6560808960, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:47:22,584] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:47:22,637] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:23,654] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:23,657] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:23,658] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:23,660] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:47:24,026] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:47:24,157] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:47:24,576] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:24,937] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:24,949] - [rackzonetests:378] INFO - verified! zone 'Group 1' is existed [2021-05-11 17:47:24,960] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:47:24,960] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 21.69405815423514, 'mem_free': 6560808960, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:47:24,961] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:47:24,963] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:47:25,095] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:47:25,506] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:25,855] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:25,859] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:47:25,992] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:47:26,448] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:26,829] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:26,832] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:47:26,940] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:47:27,250] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:27,560] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:27,563] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:47:27,694] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:47:28,104] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:28,453] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:45,446] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #7 test_check_default_zone_create_by_default ============== [2021-05-11 17:47:45,486] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:47:46,024] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:47:46,029] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:47:46,029] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:47:46,039] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:47:46,046] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:47:46,053] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:47:46,063] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:47:46,081] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:47:46,081] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:47:46,085] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:47:46,090] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:47:46,090] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:47:46,093] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:47:46,099] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:47:46,099] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:47:46,103] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:47:46,108] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:47:46,108] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:47:46,111] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:47:46,112] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #7 test_check_default_zone_create_by_default ============== [2021-05-11 17:47:46,112] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 17:47:46,112] - [basetestcase:608] INFO - closing all memcached connections [2021-05-11 17:47:46,114] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:47:46,243] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:47:46,658] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:47,011] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:47,103] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:47:47,123] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:47:47,123] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:47:47,123] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:47:47,123] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: service couchbase-server restart [2021-05-11 17:47:49,056] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:47:49,056] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:47:49,059] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:47:49,199] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:47:49,655] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:50,033] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:50,180] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:47:50,203] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:47:50,203] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:47:50,203] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:47:50,203] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: service couchbase-server restart [2021-05-11 17:47:52,199] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:47:52,199] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:47:52,202] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:47:52,303] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:47:52,597] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:52,915] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:53,025] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:47:53,039] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:47:53,039] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:47:53,039] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:47:53,039] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: service couchbase-server restart [2021-05-11 17:47:54,873] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:47:54,873] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:47:54,875] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:47:55,017] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:47:55,467] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:47:55,818] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:47:55,895] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:47:55,911] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:47:55,911] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:47:55,911] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:47:55,911] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: service couchbase-server restart [2021-05-11 17:47:57,830] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:47:57,830] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:47:57,832] - [basetestcase:1741] INFO - ========= RESET ENVIRONMENT SETTING TO ORIGINAL =========== Cluster instance shutdown with force [2021-05-11 17:47:57,834] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/nodes/self error [Errno 111] Connection refused [2021-05-11 17:48:00,840] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/nodes/self error [Errno 111] Connection refused [2021-05-11 17:48:06,859] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:06,880] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:48:07,014] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:48:07,411] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:07,771] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:07,772] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:48:07,831] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 7117 [2021-05-11 17:48:07,832] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:48:14,842] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:48:14,978] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:48:15,452] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:15,846] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:15,847] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:48:15,963] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 3173 [2021-05-11 17:48:15,964] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:48:22,973] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:48:23,075] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:48:23,372] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:23,698] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:23,699] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:48:23,793] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 26371 [2021-05-11 17:48:23,793] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:48:30,802] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:48:30,938] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:48:31,359] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:31,710] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:31,711] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:48:31,811] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 27270 [2021-05-11 17:48:31,812] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_7 ok ---------------------------------------------------------------------- Ran 1 test in 113.565s OK test_create_zone_with_upper_lower_number_and_space_name (rackzone.rackzonetests.RackzoneTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_8 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t rackzone.rackzonetests.RackzoneTests.test_create_zone_with_upper_lower_number_and_space_name,items=1000 Test Input params: {'items': '1000', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 8, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_8'} [2021-05-11 17:48:38,846] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:48:38,986] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:48:39,399] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:39,766] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:39,774] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:39,787] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:48:39,798] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:48:39,798] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #8 test_create_zone_with_upper_lower_number_and_space_name============== [2021-05-11 17:48:39,818] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:48:39,828] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:48:39,837] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:48:39,853] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:48:39,858] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:39,876] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:48:39,876] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:39,882] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:48:39,887] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:48:39,888] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:39,892] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:48:39,897] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:48:39,897] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:39,900] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:48:39,905] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:48:39,905] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:48:39,909] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:48:39,909] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:48:40,852] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:48:40,857] - [task:157] INFO - {'uptime': '46', 'memoryTotal': 8192827392, 'memoryFree': 6811820032, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3419, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 1000} [2021-05-11 17:48:40,862] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 17:48:40,862] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:48:40,862] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:48:40,873] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 17:48:40,874] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:48:40,874] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:48:40,883] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 17:48:40,895] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 17:48:40,895] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 17:48:40,899] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:48:40,899] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:48:40,899] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:48:40,900] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:48:40,993] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:48:40,995] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:48:41,126] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:48:41,519] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:41,859] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:41,872] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:48:41,933] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:48:41,933] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:48:41,936] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:41,940] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:41,948] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:48:41,979] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:48:41,990] - [task:157] INFO - {'uptime': '46', 'memoryTotal': 8191283200, 'memoryFree': 6979551232, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4186, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:48:42,002] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:48:42,015] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:48:42,015] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:48:42,104] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:48:42,106] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:48:42,248] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:48:42,711] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:43,080] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:43,093] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:48:43,159] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:48:43,159] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:48:43,163] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:43,166] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:43,172] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:48:43,202] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:48:43,207] - [task:157] INFO - {'uptime': '41', 'memoryTotal': 7934996480, 'memoryFree': 6845095936, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4055, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:48:43,212] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:48:43,222] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:48:43,222] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:48:43,307] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:48:43,309] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:48:43,414] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:48:43,688] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:44,006] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:44,016] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:48:44,220] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:48:44,221] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:48:44,224] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:44,226] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:44,232] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:48:44,251] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:48:44,255] - [task:157] INFO - {'uptime': '41', 'memoryTotal': 8192131072, 'memoryFree': 6987710464, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:48:44,261] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:48:44,270] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:48:44,270] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:48:44,363] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:48:44,365] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:48:44,494] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:48:44,901] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:45,232] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:45,244] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:48:45,304] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:48:45,304] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:48:45,307] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:45,309] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:48:45,315] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:48:45,344] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:48:45,460] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:48:50,466] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:48:50,482] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:48:50,484] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:48:50,616] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:48:51,002] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:51,352] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:51,362] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:48:51,425] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:48:51,426] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:48:51,426] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:48:52,364] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3419&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:48:52,394] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 17:48:52,395] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:48:53,025] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,132] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,188] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:48:53,188] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:48:53,292] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,379] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,455] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:48:53,455] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:48:53,558] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,689] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,749] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:48:53,749] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:48:53,879] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:53,986] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:54,064] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:48:54,064] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:48:54,148] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:54,257] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:54,324] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:48:54,325] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:48:54,473] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:54,568] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:48:54,641] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:48:54,643] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #8 test_create_zone_with_upper_lower_number_and_space_name ============== [2021-05-11 17:48:54,654] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:48:54,797] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:48:55,197] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:55,563] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:55,567] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:48:55,715] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:48:56,174] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:56,548] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:56,551] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:48:56,653] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:48:56,944] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:57,251] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:48:57,255] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:48:57,384] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:48:57,795] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:48:58,143] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:14,637] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:49:14,637] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 27.78059928897918, 'mem_free': 6696103936, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:49:14,637] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:49:14,754] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:49:15,742] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:49:15,743] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:49:15,745] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:49:15,746] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:49:16,071] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:49:16,196] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:49:16,575] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:16,926] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:16,935] - [rackzonetests:103] INFO - create zone AAAB BBCCC aakkk kmmm3 456 72 [2021-05-11 17:49:16,962] - [rest_client:4067] INFO - zone AAAB BBCCC aakkk kmmm3 456 72 is added [2021-05-11 17:49:16,971] - [rackzonetests:378] INFO - verified! zone 'AAAB BBCCC aakkk kmmm3 456 72' is existed [2021-05-11 17:49:16,983] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:49:16,983] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 27.78059928897918, 'mem_free': 6696103936, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:49:16,983] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:49:16,985] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:49:17,119] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:49:17,506] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:17,867] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:17,869] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:49:18,007] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:49:18,463] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:18,822] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:18,824] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:49:18,930] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:49:19,221] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:19,537] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:19,541] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:49:19,704] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:49:20,224] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:20,653] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:38,054] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #8 test_create_zone_with_upper_lower_number_and_space_name ============== [2021-05-11 17:49:38,087] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:49:38,656] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:49:38,660] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:49:38,660] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:49:38,669] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:49:38,676] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:49:38,683] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:49:38,689] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:49:38,704] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:49:38,704] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:49:38,708] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:49:38,712] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:49:38,712] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:49:38,716] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:49:38,721] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:49:38,721] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:49:38,725] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:49:38,730] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:49:38,730] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:49:38,734] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:49:38,734] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #8 test_create_zone_with_upper_lower_number_and_space_name ============== [2021-05-11 17:49:38,734] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 17:49:38,735] - [basetestcase:608] INFO - closing all memcached connections [2021-05-11 17:49:38,736] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:49:38,865] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:49:39,267] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:39,622] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:39,733] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:49:39,763] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:49:39,763] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:49:39,764] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:49:39,764] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: service couchbase-server restart [2021-05-11 17:49:41,689] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:49:41,689] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:49:41,692] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:49:41,858] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:49:42,370] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:42,841] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:43,076] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:49:43,113] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:49:43,113] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:49:43,113] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:49:43,113] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: service couchbase-server restart [2021-05-11 17:49:45,051] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:49:45,052] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:49:45,055] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:49:45,180] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:49:45,492] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:45,839] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:46,006] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:49:46,028] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:49:46,029] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:49:46,029] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:49:46,029] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: service couchbase-server restart [2021-05-11 17:49:47,806] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:49:47,807] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:49:47,810] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:49:48,023] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:49:48,493] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:49:48,916] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:49:49,115] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: mv /opt/couchbase/bin/couchbase-server.bak /opt/couchbase/bin/couchbase-server [2021-05-11 17:49:49,140] - [remote_util:3473] INFO - command executed with root but got an error ['mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory'] ... [2021-05-11 17:49:49,140] - [remote_util:3222] ERROR - mv: cannot stat ‘/opt/couchbase/bin/couchbase-server.bak’: No such file or directory [2021-05-11 17:49:49,140] - [remote_util:4228] INFO - this node is centos 7.x [2021-05-11 17:49:49,140] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: service couchbase-server restart [2021-05-11 17:49:50,997] - [remote_util:3473] INFO - command executed with root but got an error ['Redirecting to /bin/systemctl restart couchbase-server.service'] ... [2021-05-11 17:49:50,997] - [remote_util:3222] ERROR - Redirecting to /bin/systemctl restart couchbase-server.service [2021-05-11 17:49:50,998] - [basetestcase:1741] INFO - ========= RESET ENVIRONMENT SETTING TO ORIGINAL =========== Cluster instance shutdown with force [2021-05-11 17:49:51,004] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/nodes/self error [Errno 111] Connection refused [2021-05-11 17:49:54,008] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.105.151:8091/nodes/self error [Errno 111] Connection refused [2021-05-11 17:50:00,051] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:00,096] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:50:00,273] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:50:00,783] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:01,221] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:01,222] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:50:01,298] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 12028 [2021-05-11 17:50:01,298] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:50:08,303] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:50:08,442] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:50:08,916] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:09,288] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:09,289] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:50:09,402] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 6699 [2021-05-11 17:50:09,402] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:50:16,411] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:50:16,595] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:50:16,946] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:17,301] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:17,302] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:50:17,372] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 30326 [2021-05-11 17:50:17,373] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:50:24,383] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:50:24,558] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:50:25,061] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:25,486] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:25,487] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:50:25,592] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 31432 [2021-05-11 17:50:25,593] - [basetestcase:662] INFO - sleep for 7 secs. Time needed for couchbase server starts completely. ... [2021-05-11 17:50:32,646] - [rest_client:4086] INFO - zone AAAB BBCCC aakkk kmmm3 456 72 is deleted summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_8 ok ---------------------------------------------------------------------- Ran 1 test in 113.803s OK test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_9 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=update,test_with_view=True,num_ddocs=2,num_views_per_ddoc=3,items=1000,sasl_buckets=1,standard_buckets=1,nodes_init=1 Test Input params: {'ddoc_ops': 'update', 'test_with_view': 'True', 'num_ddocs': '2', 'num_views_per_ddoc': '3', 'items': '1000', 'sasl_buckets': '1', 'standard_buckets': '1', 'nodes_init': '1', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 9, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_9'} [2021-05-11 17:50:32,670] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:50:32,807] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:50:33,205] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:33,561] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:33,568] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:33,580] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:50:33,589] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:50:33,589] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #9 test_view_ops============== [2021-05-11 17:50:33,606] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:50:33,631] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:50:33,640] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:50:33,667] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:50:33,677] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:33,693] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:50:33,693] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:33,697] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:50:33,710] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:50:33,710] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:33,722] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:50:33,726] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:50:33,726] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:33,730] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:50:33,735] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:50:33,735] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:50:33,739] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:50:33,739] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:50:34,677] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:50:34,683] - [task:157] INFO - {'uptime': '48', 'memoryTotal': 8192827392, 'memoryFree': 6816772096, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3419, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 1000} [2021-05-11 17:50:34,688] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 17:50:34,688] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:50:34,688] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:50:34,699] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 17:50:34,700] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:50:34,700] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:50:34,713] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 17:50:34,726] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 17:50:34,726] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 17:50:34,730] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:50:34,731] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:50:34,731] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:50:34,731] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:50:34,830] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:50:34,832] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:50:34,976] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:50:35,383] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:35,754] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:35,767] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:50:35,828] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:50:35,828] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:50:35,832] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:35,834] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:35,850] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:50:35,889] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:50:35,894] - [task:157] INFO - {'uptime': '47', 'memoryTotal': 8191283200, 'memoryFree': 6959616000, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4186, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:50:35,899] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:50:35,907] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:50:35,908] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:50:36,009] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:50:36,012] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:50:36,151] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:50:36,610] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:36,985] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:36,999] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:50:37,061] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:50:37,062] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:50:37,065] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:37,068] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:37,074] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:50:37,096] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:50:37,101] - [task:157] INFO - {'uptime': '42', 'memoryTotal': 7934996480, 'memoryFree': 6848937984, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4055, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:50:37,106] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:50:37,115] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:50:37,115] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:50:37,208] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:50:37,211] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:50:37,310] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:50:37,590] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:37,901] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:37,911] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:50:38,116] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:50:38,116] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:50:38,119] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:38,122] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:38,127] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:50:38,146] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:50:38,150] - [task:157] INFO - {'uptime': '42', 'memoryTotal': 8192131072, 'memoryFree': 6992949248, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:50:38,155] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:50:38,164] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:50:38,164] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:50:38,250] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:50:38,252] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:50:38,382] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:50:38,811] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:39,161] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:39,171] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:50:39,233] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:50:39,233] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:50:39,237] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:39,239] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:50:39,247] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:50:39,277] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:50:39,398] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:50:44,403] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:50:44,419] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:50:44,422] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:50:44,576] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:50:45,052] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:45,420] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:45,433] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:50:45,498] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:50:45,498] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:50:45,499] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:50:46,346] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=1139&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:50:46,385] - [rest_client:2843] INFO - 0.04 seconds to create bucket default [2021-05-11 17:50:46,385] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:50:47,123] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:47,226] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:47,284] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:50:47,284] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:50:47,403] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:47,529] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:47,587] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:50:47,588] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:50:47,688] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:47,824] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:47,884] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:50:47,884] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:50:48,020] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:48,132] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:48,193] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:50:48,193] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:50:48,336] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:48,471] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:48,537] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:50:48,537] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:50:48,652] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:48,770] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:50:48,860] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:50:49,921] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=bucket0&ramQuotaMB=1139&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:50:49,961] - [rest_client:2843] INFO - 0.04 seconds to create bucket bucket0 [2021-05-11 17:50:49,961] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:50,807] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:51,131] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:51,291] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:50:51,291] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:51,588] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:51,914] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:52,076] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:50:52,076] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:52,311] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:52,602] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:52,756] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:50:52,757] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:53,057] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:53,333] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:53,450] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:50:53,451] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:53,714] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:53,963] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:54,080] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:50:54,081] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:54,293] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:54,497] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:50:54,654] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:50:55,685] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=standard_bucket0&ramQuotaMB=1139&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:50:55,718] - [rest_client:2843] INFO - 0.03 seconds to create bucket standard_bucket0 [2021-05-11 17:50:55,719] - [bucket_helper:335] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:56,394] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:56,529] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:56,602] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:50:56,602] - [bucket_helper:335] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:56,756] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:56,999] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:57,063] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:50:57,063] - [bucket_helper:335] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:57,152] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:57,304] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:57,407] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:50:57,408] - [bucket_helper:335] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:57,612] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:57,813] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:57,953] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:50:57,953] - [bucket_helper:335] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:58,123] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:58,318] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:58,407] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:50:58,408] - [bucket_helper:335] INFO - waiting for memcached bucket : standard_bucket0 in 172.23.105.151 to accept set ops [2021-05-11 17:50:58,613] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:58,764] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:50:58,882] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:50:58,882] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #9 test_view_ops ============== [2021-05-11 17:50:58,906] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:50:59,062] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:50:59,523] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:50:59,933] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:50:59,936] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:51:00,101] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:51:00,629] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:51:01,081] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:51:01,084] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:51:01,260] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:51:01,625] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:51:02,022] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:51:02,024] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:51:02,187] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:51:02,662] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:51:03,064] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:51:20,969] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:51:20,969] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 15.88548264504687, 'mem_free': 6527582208, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:51:20,969] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:51:20,969] - [basetestcase:1413] INFO - create 1000 documents... [2021-05-11 17:51:21,061] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:51:21,256] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:51:21,381] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:51:22,080] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:51:22,080] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:51:22,093] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:51:22,093] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:51:22,416] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:51:22,416] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:51:22,421] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:51:22,434] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:51:22,763] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:51:22,767] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:51:22,770] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:51:22,783] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:51:23,130] - [createdeleteview:78] INFO - Processing Create DDoc Operation On Bucket default [2021-05-11 17:51:24,019] - [rest_client:1022] ERROR - GET http://172.23.105.151:8092//default/_design/dev_ddoc0 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2021-05-11 17:51:24,020] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:24,062] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views0?stale=ok [2021-05-11 17:51:24,176] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:24,248] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:24,297] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views1?stale=ok [2021-05-11 17:51:24,380] - [task:2844] INFO - view : views1 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:24,450] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:24,519] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views2?stale=ok [2021-05-11 17:51:29,557] - [task:2844] INFO - view : views2 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:29,630] - [rest_client:1022] ERROR - GET http://172.23.105.151:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2021-05-11 17:51:29,631] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:29,659] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views0?stale=ok [2021-05-11 17:51:34,709] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:51:34,775] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:34,823] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views1?stale=ok [2021-05-11 17:51:34,902] - [task:2844] INFO - view : views1 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:51:34,971] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:35,039] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views2?stale=ok [2021-05-11 17:51:35,069] - [task:2844] INFO - view : views2 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:51:35,078] - [createdeleteview:95] INFO - Processing Update DDoc Operation On Bucket default [2021-05-11 17:51:35,144] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:35,217] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views0?stale=ok [2021-05-11 17:51:40,262] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:40,266] - [createdeleteview:78] INFO - Processing Create DDoc Operation On Bucket bucket0 [2021-05-11 17:51:40,335] - [rest_client:1022] ERROR - GET http://172.23.105.151:8092//bucket0/_design/dev_ddoc0 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2021-05-11 17:51:40,336] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:40,369] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views0?stale=ok [2021-05-11 17:51:40,446] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:40,524] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:40,579] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views1?stale=ok [2021-05-11 17:51:40,656] - [task:2844] INFO - view : views1 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:40,738] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:40,812] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views2?stale=ok [2021-05-11 17:51:40,866] - [task:2844] INFO - view : views2 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:40,961] - [rest_client:1022] ERROR - GET http://172.23.105.151:8092//bucket0/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2021-05-11 17:51:40,962] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:40,998] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views0?stale=ok [2021-05-11 17:51:46,045] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:51:46,132] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:46,186] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views1?stale=ok [2021-05-11 17:51:51,238] - [task:2844] INFO - view : views1 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:51:51,326] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:51,402] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views2?stale=ok [2021-05-11 17:51:51,433] - [task:2844] INFO - view : views2 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:51:51,445] - [createdeleteview:95] INFO - Processing Update DDoc Operation On Bucket bucket0 [2021-05-11 17:51:51,514] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:51,591] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views0?stale=ok [2021-05-11 17:51:56,627] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:51:56,641] - [createdeleteview:78] INFO - Processing Create DDoc Operation On Bucket standard_bucket0 [2021-05-11 17:51:56,715] - [rest_client:1022] ERROR - GET http://172.23.105.151:8092//standard_bucket0/_design/dev_ddoc0 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2021-05-11 17:51:56,716] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:51:56,763] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views0?stale=ok [2021-05-11 17:52:01,823] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:52:01,910] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:52:01,967] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views1?stale=ok [2021-05-11 17:52:07,016] - [task:2844] INFO - view : views1 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:52:07,105] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:52:07,179] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views2?stale=ok [2021-05-11 17:52:12,211] - [task:2844] INFO - view : views2 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:52:12,291] - [rest_client:1022] ERROR - GET http://172.23.105.151:8092//standard_bucket0/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2021-05-11 17:52:12,291] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:52:12,325] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views0?stale=ok [2021-05-11 17:52:17,393] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:52:17,471] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:52:17,527] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views1?stale=ok [2021-05-11 17:52:22,583] - [task:2844] INFO - view : views1 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:52:22,678] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:52:22,755] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views2?stale=ok [2021-05-11 17:52:22,788] - [task:2844] INFO - view : views2 was created successfully in ddoc: dev_ddoc1 [2021-05-11 17:52:22,801] - [createdeleteview:95] INFO - Processing Update DDoc Operation On Bucket standard_bucket0 [2021-05-11 17:52:22,883] - [rest_client:503] INFO - -->create_design_document [2021-05-11 17:52:22,960] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views0?stale=ok [2021-05-11 17:52:28,000] - [task:2844] INFO - view : views0 was created successfully in ddoc: dev_ddoc0 [2021-05-11 17:52:28,036] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:52:28,056] - [basetestcase:2629] INFO - list of kv nodes in cluster: [ip:172.23.105.151 port:8091 ssh_username:root] [2021-05-11 17:52:28,095] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:52:28,227] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 17:52:28,250] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 17:52:28,348] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',bucket0 bucket [2021-05-11 17:52:28,372] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 standard_bucket0 [2021-05-11 17:52:28,468] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',standard_bucket0 bucket [2021-05-11 17:52:28,468] - [createdeleteview:241] INFO - DDoc Validation Started [2021-05-11 17:52:28,483] - [createdeleteview:249] INFO - Database Document dev_ddoc0 details : {"views": {"views0": {"map": "function (doc) { emit(null, doc);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views2": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} [2021-05-11 17:52:28,486] - [createdeleteview:249] INFO - Database Document dev_ddoc1 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views2": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} [2021-05-11 17:52:28,489] - [createdeleteview:249] INFO - Database Document dev_ddoc0 details : {"views": {"views0": {"map": "function (doc) { emit(null, doc);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views2": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} [2021-05-11 17:52:28,493] - [createdeleteview:249] INFO - Database Document dev_ddoc1 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views2": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} [2021-05-11 17:52:28,496] - [createdeleteview:249] INFO - Database Document dev_ddoc0 details : {"views": {"views0": {"map": "function (doc) { emit(null, doc);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views2": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} [2021-05-11 17:52:28,499] - [createdeleteview:249] INFO - Database Document dev_ddoc1 details : {"views": {"views0": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views1": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}, "views2": {"map": "function (doc) { emit(doc.age, doc.first_name);}"}}} [2021-05-11 17:52:28,499] - [createdeleteview:258] INFO - DDoc Validation Successful [2021-05-11 17:52:28,517] - [createdeleteview:272] INFO - DDoc Data Validation Started on bucket default. Expected Data Items 1000 [2021-05-11 17:52:29,485] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:30,829] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:30,860] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views0, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:30,860] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:31,874] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:31,912] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:31,969] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views1, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:31,969] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:32,986] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:33,020] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc0/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:33,041] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views2, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:33,041] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:34,057] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:35,159] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:35,185] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views0, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:35,186] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:36,202] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:36,236] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:36,256] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views1, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:36,256] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:37,274] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:37,310] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/default/_design/dev_ddoc1/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:37,333] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views2, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:37,333] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:37,338] - [createdeleteview:272] INFO - DDoc Data Validation Started on bucket bucket0. Expected Data Items 1000 [2021-05-11 17:52:38,348] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:39,506] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:39,531] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views0, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:39,531] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:40,546] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:40,582] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:40,603] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views1, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:40,603] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:41,611] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:41,636] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc0/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:41,658] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views2, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:41,658] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:42,666] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:43,996] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:44,034] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views0, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:44,034] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:45,042] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:45,084] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:45,119] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views1, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:45,120] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:46,127] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:46,170] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/bucket0/_design/dev_ddoc1/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:46,206] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views2, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:46,207] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:46,211] - [createdeleteview:272] INFO - DDoc Data Validation Started on bucket standard_bucket0. Expected Data Items 1000 [2021-05-11 17:52:47,214] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:48,560] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:48,604] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views0, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:48,605] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:49,613] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:49,650] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:49,685] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views1, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:49,685] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:50,695] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:50,735] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc0/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:50,769] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc0, View: views2, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:50,769] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:51,778] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:52,909] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views0?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:52,941] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views0, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:52,942] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:53,950] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:53,988] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views1?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:54,021] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views1, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:54,021] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:55,037] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:55,086] - [rest_client:565] INFO - index query url: http://172.23.105.151:8092/standard_bucket0/_design/dev_ddoc1/_view/views2?stale=false&full_set=true&connection_timeout=60000 [2021-05-11 17:52:55,117] - [task:3056] INFO - Server: 172.23.105.151, Design Doc: dev_ddoc1, View: views2, (1000 rows) expected, (1000 rows) returned [2021-05-11 17:52:55,118] - [task:3063] INFO - expected number of rows: '1000' was found for view query [2021-05-11 17:52:55,118] - [createdeleteview:278] INFO - DDoc Data Validation Successful [2021-05-11 17:52:55,148] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:52:55,148] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 15.34942820838628, 'mem_free': 6482935808, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:52:55,148] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:52:55,151] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:52:55,327] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:52:55,816] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:52:56,239] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:52:56,241] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:52:56,400] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:52:56,938] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:52:57,421] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:52:57,425] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:52:57,591] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:52:57,943] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:52:58,343] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:52:58,346] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:52:58,479] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:52:58,924] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:52:59,279] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:17,040] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #9 test_view_ops ============== [2021-05-11 17:53:17,112] - [bucket_helper:133] INFO - deleting existing buckets ['bucket0', 'default', 'standard_bucket0'] on 172.23.105.151 [2021-05-11 17:53:18,675] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:53:18,711] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['default', 'standard_bucket0'] [2021-05-11 17:53:18,713] - [bucket_helper:156] INFO - deleted bucket : bucket0 from 172.23.105.151 [2021-05-11 17:53:20,355] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:53:20,416] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['standard_bucket0'] [2021-05-11 17:53:20,417] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:53:21,387] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:53:21,398] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:53:21,399] - [bucket_helper:156] INFO - deleted bucket : standard_bucket0 from 172.23.105.151 [2021-05-11 17:53:21,411] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:53:21,433] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:53:21,445] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:53:21,458] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:21,504] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:53:21,504] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:21,517] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:53:21,522] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:53:21,522] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:21,526] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:53:21,538] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:53:21,538] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:21,549] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:53:21,553] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:53:21,553] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:21,557] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:53:21,557] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #9 test_view_ops ============== [2021-05-11 17:53:21,557] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 17:53:21,558] - [basetestcase:608] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_9 ok ---------------------------------------------------------------------- Ran 1 test in 168.924s OK test_backup_restore_sanity (ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_10 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest.test_backup_restore_sanity,items=1000,reset_services=True Test Input params: {'items': '1000', 'reset_services': 'True', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 10, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_10'} [2021-05-11 17:53:22,221] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:53:22,423] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:53:22,883] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:23,325] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:23,342] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:23,374] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:53:23,385] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 17:53:23,385] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #10 test_backup_restore_sanity============== [2021-05-11 17:53:23,402] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 17:53:23,416] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:53:23,423] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 17:53:23,430] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 17:53:23,436] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:23,455] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:53:23,456] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:23,459] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:53:23,467] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:53:23,467] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:23,474] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:53:23,479] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 17:53:23,479] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:23,483] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 17:53:23,488] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 17:53:23,488] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:53:23,493] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 17:53:23,493] - [basetestcase:295] INFO - initializing cluster [2021-05-11 17:53:23,495] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:53:23,639] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:53:24,085] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:24,530] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:24,560] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:53:24,727] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:53:25,169] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:25,609] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:25,669] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:53:25,842] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 17:53:25,843] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service [2021-05-11 17:53:27,870] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:27,871] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 17:53:27,871] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:53:32,874] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 17:53:32,914] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:32,914] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 17:53:32,991] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:32,994] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:53:33,172] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:53:33,684] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:34,136] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:34,239] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:53:34,240] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:53:34,310] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 17:53:34,512] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 17:53:34,512] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service [2021-05-11 17:53:34,557] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:34,557] - [remote_util:338] INFO - 172.23.105.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 17:53:39,563] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 17:53:39,629] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:39,629] - [remote_util:4006] INFO - Couchbase server status: [' Process: 17162 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 17:53:39,630] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:53:39,721] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 17364 [2021-05-11 17:53:39,721] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 17:53:39,723] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:53:39,901] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:53:40,396] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:40,833] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:40,861] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:53:41,030] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:53:41,560] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:42,028] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:42,094] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:53:42,288] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 17:53:42,288] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service [2021-05-11 17:53:44,222] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:44,222] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 17:53:44,223] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:53:49,228] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 17:53:49,253] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:49,253] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 17:53:49,321] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:49,324] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:53:49,464] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:53:50,009] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:50,455] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:50,486] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:53:50,486] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:53:50,548] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 17:53:50,739] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 17:53:50,740] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service [2021-05-11 17:53:50,783] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:50,783] - [remote_util:338] INFO - 172.23.105.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 17:53:55,788] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 17:53:55,831] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:55,832] - [remote_util:4006] INFO - Couchbase server status: [' Process: 10504 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 17:53:55,832] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:53:55,919] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 10682 [2021-05-11 17:53:55,919] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 17:53:55,922] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:53:56,064] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:53:56,402] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:56,787] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:56,815] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:53:56,986] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:53:57,332] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:53:57,900] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:53:57,944] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:53:58,076] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 17:53:58,077] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl stop couchbase-server.service [2021-05-11 17:53:59,822] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:53:59,823] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 17:53:59,823] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:54:04,827] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 17:54:04,853] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:04,853] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 17:54:04,922] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:04,924] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:54:05,091] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:54:05,439] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:05,836] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:05,890] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:54:05,890] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:54:05,961] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 17:54:06,132] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 17:54:06,133] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl start couchbase-server.service [2021-05-11 17:54:06,177] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:06,178] - [remote_util:338] INFO - 172.23.107.160:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 17:54:11,183] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 17:54:11,231] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:11,231] - [remote_util:4006] INFO - Couchbase server status: [' Process: 2056 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 17:54:11,231] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:54:11,321] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 2246 [2021-05-11 17:54:11,321] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 17:54:11,324] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:54:11,516] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:54:12,002] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:12,419] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:12,446] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:54:12,645] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:54:13,090] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:13,482] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:13,531] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:54:13,679] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 17:54:13,679] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl stop couchbase-server.service [2021-05-11 17:54:16,430] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:16,430] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 17:54:16,430] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:54:21,436] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 17:54:21,474] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:21,474] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 17:54:21,544] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:21,547] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:54:21,751] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:54:22,227] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:22,649] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:22,705] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:54:22,706] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:54:22,769] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 17:54:22,983] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 17:54:22,984] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl start couchbase-server.service [2021-05-11 17:54:23,024] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:23,025] - [remote_util:338] INFO - 172.23.107.184:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 17:54:28,030] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 17:54:28,074] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:28,074] - [remote_util:4006] INFO - Couchbase server status: [' Process: 3476 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 17:54:28,075] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:54:28,169] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 3626 [2021-05-11 17:54:28,170] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 17:54:28,170] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 17:54:38,311] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 17:54:38,315] - [task:157] INFO - {'uptime': '57', 'memoryTotal': 8192827392, 'memoryFree': 6909968384, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4022, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 17:54:38,320] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:54:38,320] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 17:54:38,320] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:54:38,320] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:54:38,330] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 17:54:38,330] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:54:38,331] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:54:38,335] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 17:54:38,340] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 17:54:38,341] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 17:54:38,372] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:54:38,373] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:54:38,476] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:54:38,479] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:54:38,610] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:54:39,032] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:39,412] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:39,429] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:54:39,491] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:39,491] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:54:39,495] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:39,497] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:39,505] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:54:39,545] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 17:54:39,554] - [task:157] INFO - {'uptime': '43', 'memoryTotal': 8191283200, 'memoryFree': 6963068928, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 17:54:39,559] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:54:39,559] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 17:54:39,570] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:54:39,571] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 17:54:39,673] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:54:39,675] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:54:39,826] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:54:40,318] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:40,697] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:40,710] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:54:40,775] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:40,776] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:54:40,779] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:40,784] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:40,789] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:54:40,817] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 17:54:40,828] - [task:157] INFO - {'uptime': '27', 'memoryTotal': 7934996480, 'memoryFree': 6852386816, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 17:54:40,837] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:54:40,838] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 17:54:40,853] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:54:40,853] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 17:54:40,958] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:54:40,959] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:54:41,059] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:54:41,347] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:41,666] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:41,696] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:54:41,881] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:41,881] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:54:41,892] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:41,901] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:41,913] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:54:41,965] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 17:54:41,976] - [task:157] INFO - {'uptime': '12', 'memoryTotal': 8192131072, 'memoryFree': 6976192512, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 17:54:41,986] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:54:41,986] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 17:54:42,001] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:54:42,001] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 17:54:42,121] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:54:42,123] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:54:42,335] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:54:42,817] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:43,237] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:43,248] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:54:43,313] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:43,314] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:54:43,318] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:43,323] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 17:54:43,329] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 17:54:43,362] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:54:43,378] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 17:54:43,381] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 17:54:43,493] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:54:48,498] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:54:48,513] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 17:54:48,515] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:54:48,641] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:54:49,050] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:49,394] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:49,407] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:54:49,468] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:54:49,469] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:54:49,469] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 17:54:50,381] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3419&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:54:50,420] - [rest_client:2843] INFO - 0.04 seconds to create bucket default [2021-05-11 17:54:50,421] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:54:51,212] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:51,398] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:51,558] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 17:54:51,558] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:54:51,756] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:52,000] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:52,117] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 17:54:52,117] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:54:52,351] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:52,623] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:52,777] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 17:54:52,777] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:54:52,978] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:53,214] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:53,331] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 17:54:53,332] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:54:53,581] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:53,768] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:53,924] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 17:54:53,925] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 17:54:54,113] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:54,340] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:54:54,460] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 17:54:54,463] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #10 test_backup_restore_sanity ============== [2021-05-11 17:54:54,473] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:54:54,607] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:54:55,003] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:55,368] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:55,372] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:54:55,530] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:54:55,984] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:56,358] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:56,361] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:54:56,462] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:54:56,756] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:57,079] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:54:57,082] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 17:54:57,214] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 17:54:57,632] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:54:57,976] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:15,128] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:55:15,128] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 30.4093567251462, 'mem_free': 6722490368, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:55:15,128] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:55:15,128] - [newupgradebasetest:55] INFO - ============== NewUpgradeBaseTest setup has started ============== [2021-05-11 17:55:15,130] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:55:15,262] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:55:15,659] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:16,017] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:16,019] - [newupgradebasetest:218] INFO - ============== NewUpgradeBaseTest setup has completed ============== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 52 100 20 100 32 7396 11834 --:--:-- --:--:-- --:--:-- 16000 [2021-05-11 17:55:16,040] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:55:16,172] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:55:16,551] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:16,908] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:16,918] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:55:16,920] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:17,020] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:17,310] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:17,626] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:17,627] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: nproc [2021-05-11 17:55:17,648] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:17,723] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:17,820] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:18,098] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:18,411] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:18,478] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:55:18,608] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:55:19,001] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:19,357] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:19,467] - [enterprise_backup_restore_test:83] INFO - *** start to load items to all buckets [2021-05-11 17:55:19,582] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:20,189] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:20,191] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:20,200] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:20,204] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:20,513] - [enterprise_backup_restore_test:85] INFO - *** done to load items to all buckets [2021-05-11 17:55:20,525] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:55:20,528] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:20,630] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:20,921] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:21,233] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:21,234] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /tmp/entbackup_172.23.105.151 [2021-05-11 17:55:21,288] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:21,288] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr config --archive /tmp/entbackup_172.23.105.151 --repo backup [2021-05-11 17:55:21,371] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:21,371] - [remote_util:3260] INFO - Backup repository `backup` created successfully in archive `/tmp/entbackup_172.23.105.151` [2021-05-11 17:55:21,374] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:21,472] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:21,754] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:22,070] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:22,071] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -R /tmp/entbackup_172.23.105.151/backup [2021-05-11 17:55:22,124] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:22,127] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:22,221] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:22,502] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:22,810] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:22,874] - [enterprise_backup_restore_base:573] INFO - Expected Backup Meta data json and Actual Backup Meta data json are equal Backup create validation success. [2021-05-11 17:55:22,874] - [enterprise_backup_restore_test:95] INFO - *** start to update items in all buckets [2021-05-11 17:55:22,925] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:23,504] - [enterprise_backup_restore_test:97] INFO - *** done update items in all buckets [2021-05-11 17:55:23,505] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 17:55:33,515] - [enterprise_backup_restore_test:103] INFO - *** start to validate backup cluster [2021-05-11 17:55:33,522] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:55:33,532] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:55:33,535] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:33,634] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:33,917] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:34,224] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:34,225] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --no-progress-bar [2021-05-11 17:55:37,734] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:37,734] - [enterprise_backup_restore_base:2045] INFO - Found 'Backup completed successfully in CLI output [2021-05-11 17:55:37,735] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -tr /tmp/entbackup_172.23.105.151/backup/ | grep -P '^\d{4}-\d{2}-\d{2}' | tail -1 [2021-05-11 17:55:37,789] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:37,790] - [enterprise_backup_restore_base:675] INFO - Finished taking backup with args: backup --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --no-progress-bar [2021-05-11 17:55:37,791] - [enterprise_backup_restore_base:2045] INFO - Found 'Backup completed successfully in CLI output [2021-05-11 17:55:37,799] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:55:37,801] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:37,894] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:38,188] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:38,499] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:38,500] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr info --archive /tmp/entbackup_172.23.105.151 --all --json [2021-05-11 17:55:38,596] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:38,599] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:38,732] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:39,015] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:39,325] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:39,326] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -lR /tmp/entbackup_172.23.105.151/backup [2021-05-11 17:55:39,446] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:39,490] - [enterprise_backup_restore_base:698] INFO - Expected Directory Structure and Actual Directory Structure are equal [2021-05-11 17:55:39,528] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:55:40,051] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:55:40,187] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:55:40,597] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:40,945] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:40,946] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 17:55:42,362] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 17:55:42,549] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:55:42,677] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:55:43,077] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:43,434] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:43,435] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 17:55:44,725] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 17:55:44,938] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:55:45,037] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:55:45,316] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:45,624] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:45,633] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: cat /tmp/entbackup_172.23.105.151/backup/2021-05-11T17_55_34.335991099-07_00/default-abaecf9a8cdb18ea5d6188cf349df164/range.json [2021-05-11 17:55:45,706] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:45,715] - [enterprise_backup_restore_test:111] INFO - *** start to restore cluster [2021-05-11 17:55:45,715] - [enterprise_backup_restore_test:115] INFO - *** start to reset cluster [2021-05-11 17:55:45,760] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:55:46,266] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:55:46,270] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:55:46,271] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:55:46,280] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:55:46,301] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:55:46,310] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:55:46,447] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:55:46,828] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:55:47,180] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:55:47,192] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:55:47,251] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:55:47,252] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:55:47,262] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 17:55:47,274] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 17:55:48,278] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:55:50,282] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:55:52,286] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:55:54,289] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:55:56,296] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 17:56:04,315] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 17:56:04,315] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:56:04,315] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:56:04,321] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 17:56:04,321] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:56:04,321] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:56:04,329] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 17:56:04,329] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 17:56:04,337] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 17:56:04,337] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 17:56:04,368] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:56:04,368] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:56:04,459] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:56:04,459] - [enterprise_backup_restore_test:126] INFO - Done reset cluster [2021-05-11 17:56:04,459] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 17:56:14,481] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:56:14,501] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 17:56:14,502] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 17:56:14,610] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:56:19,615] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:56:19,630] - [enterprise_backup_restore_test:134] INFO - *** start restore validation [2021-05-11 17:56:19,638] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:56:19,648] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:56:19,788] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:56:20,209] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:56:20,565] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:56:20,576] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:56:20,639] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:56:20,640] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:56:20,640] - [enterprise_backup_restore_base:814] INFO - Services in restore cluster: ['backup', 'fts', 'index', 'kv', 'n1ql'] [2021-05-11 17:56:20,652] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:56:20,652] - [enterprise_backup_restore_base:848] INFO - Creating bucket default in restore host 172.23.105.151 [2021-05-11 17:56:20,662] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:56:20,698] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 17:56:20,704] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 17:56:21,706] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:56:23,708] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:56:25,711] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:56:27,715] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 17:56:29,723] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 17:56:36,729] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:56:36,736] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:56:36,869] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:56:37,265] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:56:37,610] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:56:37,618] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 17:56:37,620] - [remote_util:5217] INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2021-05-11 17:56:38,631] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 17:56:38,631] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:56:38,631] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:56:38,637] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 17:56:38,637] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:56:38,637] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:56:38,642] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 17:56:38,642] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 17:56:38,647] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 17:56:38,647] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 17:56:38,675] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:56:38,676] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:56:38,765] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:56:38,766] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 17:56:48,784] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 17:56:48,784] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:57:09,275] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:57:09,309] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:57:19,323] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 17:57:19,330] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.020703792572021 sec [2021-05-11 17:57:19,338] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:57:19,494] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:57:20,000] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:20,372] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:57:20,384] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:57:20,447] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:57:20,448] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:57:20,450] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:57:20,611] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:57:21,124] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:21,525] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:57:21,540] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 17:57:21,607] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:57:21,607] - [remote_util:5220] INFO - ['ok'] [2021-05-11 17:57:21,608] - [rest_client:1182] INFO - settings/indexes params : storageMode=memory_optimized [2021-05-11 17:57:21,616] - [enterprise_backup_restore_base:2084] INFO - Done reset node [2021-05-11 17:57:22,628] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 17:57:22,628] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 17:57:22,628] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 17:57:22,635] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 17:57:22,635] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 17:57:22,635] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 17:57:22,644] - [rest_client:1099] INFO - quota for kv: 3418 MB [2021-05-11 17:57:22,644] - [rest_client:1147] INFO - pools/default params : memoryQuota=3418 [2021-05-11 17:57:22,651] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 17:57:22,651] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 17:57:22,655] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 17:57:22,656] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 17:57:22,656] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 17:57:22,656] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 17:57:22,777] - [rest_client:1052] INFO - --> status:True [2021-05-11 17:57:22,778] - [enterprise_backup_restore_base:868] INFO - replica in bucket default is 1 [2021-05-11 17:57:22,806] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=2392&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 17:57:22,875] - [rest_client:2843] INFO - 0.07 seconds to create bucket default [2021-05-11 17:57:23,563] - [basetestcase:662] INFO - sleep for 5 secs. wait for bucket is up ... [2021-05-11 17:57:28,578] - [basetestcase:662] INFO - sleep for 15 secs. wait for index service ready ... [2021-05-11 17:57:43,596] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:57:43,696] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:57:43,987] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:44,294] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:57:44,295] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr restore --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --start 2021-05-11T17_55_34.335991099-07_00 --end 2021-05-11T17_55_34.335991099-07_00 --no-progress-bar [2021-05-11 17:57:46,453] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:57:46,453] - [remote_util:3260] INFO - | Transfer [2021-05-11 17:57:46,453] - [remote_util:3260] INFO - | -------- [2021-05-11 17:57:46,454] - [remote_util:3260] INFO - | Status | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 17:57:46,454] - [remote_util:3260] INFO - | Succeeded | 82.40KiB | Tue, 11 May 2021 17:57:44 -0700/s | Tue, 11 May 2021 17:57:46 -0700 | 2.042s | [2021-05-11 17:57:46,454] - [remote_util:3260] INFO - [2021-05-11 17:57:46,454] - [remote_util:3260] INFO - | Bucket [2021-05-11 17:57:46,454] - [remote_util:3260] INFO - | ------ [2021-05-11 17:57:46,455] - [remote_util:3260] INFO - | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 17:57:46,455] - [remote_util:3260] INFO - | default | Succeeded | 164.81KiB | 164.81KiB/s | Tue, 11 May 2021 17:57:44 -0700 | Tue, 11 May 2021 17:57:46 -0700 | 1.706s | [2021-05-11 17:57:46,455] - [remote_util:3260] INFO - | [2021-05-11 17:57:46,455] - [remote_util:3260] INFO - | Mutations | Deletions | Expirations | [2021-05-11 17:57:46,456] - [remote_util:3260] INFO - | --------- | --------- | ----------- | [2021-05-11 17:57:46,456] - [remote_util:3260] INFO - | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | [2021-05-11 17:57:46,456] - [remote_util:3260] INFO - | 1000 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | [2021-05-11 17:57:46,456] - [remote_util:3260] INFO - [2021-05-11 17:57:46,456] - [remote_util:3260] INFO - Restore completed successfully [2021-05-11 17:57:46,469] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:57:46,520] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 17:57:46,623] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 17:57:46,948] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:47,276] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:57:47,285] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 17:57:47,285] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: grep 'Restore completed successfully' /tmp/entbackup_172.23.105.151/logs/backup-*.log [2021-05-11 17:57:47,335] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:57:47,335] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: grep 'Transfer failed' /tmp/entbackup_172.23.105.151/logs/backup-*.log [2021-05-11 17:57:47,392] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:57:47,392] - [enterprise_backup_restore_base:1085] INFO - Finished restoring backup [2021-05-11 17:57:47,433] - [enterprise_backup_restore_base:1086] INFO - Get current vseqno on node 172.23.105.151 [2021-05-11 17:57:47,445] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 17:57:47,465] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 17:57:47,465] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 17:57:47,595] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:57:52,600] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 17:57:52,674] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 17:57:53,178] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 17:57:53,711] - [enterprise_backup_restore_base:1094] INFO - *** Start to validate the restore [2021-05-11 17:57:53,712] - [backup_restore_validations:91] INFO - backup start: 1 [2021-05-11 17:57:53,712] - [backup_restore_validations:92] INFO - backup end: 1 [2021-05-11 17:57:53,728] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:57:53,867] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:57:54,304] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:54,667] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:57:54,668] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 17:57:55,848] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 17:57:56,035] - [backup_restore_validations:131] INFO - Compare backup data in bucket default [2021-05-11 17:57:56,036] - [enterprise_backup_restore_base:1118] INFO - End Verification for vbucket sequence numbers comparison Expected default Data and Actual default Data are equal Data validation success [2021-05-11 17:57:56,037] - [newupgradebasetest:230] INFO - ============== NewUpgradeBaseTest tearDown has started ============== [2021-05-11 17:57:56,062] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 17:57:56,062] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.958091391062863, 'mem_free': 6720770048, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:57:56,063] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.252473991372748, 'mem_free': 6774116352, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 17:57:56,063] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 17:57:56,064] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:57:56,202] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:57:56,638] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:57,012] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:57:57,016] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:57:57,162] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:57:57,674] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:57:58,064] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:58:12,734] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #10 test_backup_restore_sanity ============== [2021-05-11 17:58:12,815] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 17:58:13,613] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 17:58:13,618] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 17:58:13,618] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 17:58:13,633] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 17:58:13,641] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:58:13,655] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 17:58:13,658] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 17:58:13,699] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 17:58:13,701] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 17:58:23,716] - [rest_client:1894] INFO - rebalance percentage : 53.00 % [2021-05-11 17:58:33,736] - [rest_client:1894] INFO - rebalance percentage : 53.00 % [2021-05-11 17:58:53,765] - [rest_client:1833] INFO - rebalance progress took 40.07 seconds [2021-05-11 17:58:53,766] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 17:59:03,786] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 17:59:03,792] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 17:59:03,792] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:59:03,796] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 17:59:03,802] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 17:59:03,802] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 17:59:03,807] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 17:59:03,807] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #10 test_backup_restore_sanity ============== [2021-05-11 17:59:03,807] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 17:59:03,808] - [basetestcase:608] INFO - closing all memcached connections [2021-05-11 17:59:03,810] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:59:03,940] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:59:04,339] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:59:04,689] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:59:04,703] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:59:04,830] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:59:05,214] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:59:05,569] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:59:05,638] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:59:05,713] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 17:59:05,714] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service [2021-05-11 17:59:07,659] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:07,659] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 17:59:07,660] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:59:12,665] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 17:59:12,691] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:12,691] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 17:59:12,751] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:12,754] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 17:59:12,886] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 17:59:13,291] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:59:13,650] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:59:13,718] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:59:13,718] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:59:13,810] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 17:59:13,933] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 17:59:13,933] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service [2021-05-11 17:59:13,965] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:13,966] - [remote_util:338] INFO - 172.23.105.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 17:59:18,971] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 17:59:19,013] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:19,013] - [remote_util:4006] INFO - Couchbase server status: [' Process: 23821 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 17:59:19,013] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:59:19,092] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 23993 [2021-05-11 17:59:19,093] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 17:59:19,094] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:59:19,234] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:59:19,672] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:59:20,051] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:59:20,063] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:59:20,196] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:59:20,637] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:59:21,019] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:59:21,059] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:59:21,163] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 17:59:21,163] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service [2021-05-11 17:59:23,022] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:23,023] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 17:59:23,023] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 17:59:28,028] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 17:59:28,056] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:28,057] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 17:59:28,125] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:28,127] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 17:59:28,266] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 17:59:28,727] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 17:59:29,099] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 17:59:29,172] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 17:59:29,173] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:59:29,276] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 17:59:29,406] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 17:59:29,406] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service [2021-05-11 17:59:29,441] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:29,441] - [remote_util:338] INFO - 172.23.105.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 17:59:34,446] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 17:59:34,484] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 17:59:34,485] - [remote_util:4006] INFO - Couchbase server status: [' Process: 15436 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 17:59:34,485] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 17:59:34,569] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 15614 [2021-05-11 17:59:34,570] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 17:59:34,570] - [basetestcase:662] INFO - sleep for 10 secs. ... Cluster instance shutdown with force [2021-05-11 17:59:44,592] - [basetestcase:662] INFO - sleep for 20 secs. sleep 20 seconds before run next test ... [2021-05-11 18:00:04,603] - [newupgradebasetest:269] INFO - ============== NewUpgradeBaseTest tearDown has completed ============== [2021-05-11 18:00:04,605] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:04,712] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:04,994] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:05,299] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:05,300] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /tmp/entbackup_172.23.105.151 [2021-05-11 18:00:05,511] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:05,512] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /cbqe3043/entbackup [2021-05-11 18:00:05,566] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:05,583] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:00:05,602] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:00:05,607] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:05,628] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:05,637] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:05,647] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:00:05,655] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:00:05,659] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:05,681] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:05,689] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:05,693] - [enterprise_backup_restore_base:384] INFO - delete dir /tmp/backuprestore172.23.105.151 summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_10 ok ---------------------------------------------------------------------- Ran 1 test in 403.476s OK test_basic_ssl_test (security.x509tests.x509tests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_11 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t security.x509tests.x509tests.test_basic_ssl_test,default_bucket=False,SSLtype=openssl Test Input params: {'default_bucket': 'False', 'SSLtype': 'openssl', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 11, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_11'} [2021-05-11 18:00:05,747] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:05,886] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:06,313] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:06,666] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:06,672] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:06,684] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:00:06,690] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:00:06,691] - [rest_client:2475] INFO - Node versions in cluster [] [2021-05-11 18:00:06,691] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #11 test_basic_ssl_test============== [2021-05-11 18:00:06,696] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:00:06,710] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:00:06,718] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:00:06,725] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:00:06,732] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:00:06,736] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:06,750] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:00:06,750] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:06,753] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:00:06,758] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:00:06,758] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:06,761] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:00:06,765] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:00:06,765] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:06,769] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:00:06,773] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:00:06,773] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:00:06,777] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:00:06,777] - [basetestcase:295] INFO - initializing cluster [2021-05-11 18:00:07,753] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:00:07,757] - [task:157] INFO - {'uptime': '47', 'memoryTotal': 8192827392, 'memoryFree': 6925262848, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4022, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 18:00:07,759] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:00:07,759] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 18:00:07,759] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:00:07,759] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:00:07,768] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 18:00:07,768] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:00:07,768] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:00:07,774] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 18:00:07,779] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 18:00:07,779] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 18:00:07,807] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:00:07,807] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:00:07,901] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:00:07,903] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:08,037] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:08,480] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:08,896] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:08,911] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:00:08,975] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:08,975] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:00:08,979] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:08,982] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:08,989] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:00:09,027] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:00:09,031] - [task:157] INFO - {'uptime': '37', 'memoryTotal': 8191283200, 'memoryFree': 6970335232, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:00:09,034] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:00:09,035] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 18:00:09,043] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:00:09,043] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:00:09,155] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:00:09,157] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:00:09,306] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:00:09,798] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:10,190] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:10,202] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:00:10,268] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:10,268] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:00:10,271] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:10,274] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:10,282] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:00:10,307] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:00:10,312] - [task:157] INFO - {'uptime': '362', 'memoryTotal': 7934996480, 'memoryFree': 6816346112, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4055, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:00:10,317] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:00:10,321] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:00:10,322] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:00:10,413] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:00:10,415] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:10,521] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:10,861] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:11,179] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:11,191] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:00:11,395] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:11,396] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:00:11,399] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:11,402] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:11,407] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:00:11,427] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:00:11,433] - [task:157] INFO - {'uptime': '341', 'memoryTotal': 8192131072, 'memoryFree': 6977540096, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:00:11,438] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:00:11,444] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:00:11,444] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:00:11,536] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:00:11,537] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:00:11,679] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:00:12,131] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:12,485] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:12,498] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:00:12,559] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:12,559] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:00:12,563] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:12,566] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:00:12,572] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:00:12,604] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:00:12,628] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:00:12,629] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:00:12,761] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:00:17,766] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:00:17,781] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 18:00:17,782] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:17,916] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:18,340] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:18,698] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:18,707] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:00:18,772] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:18,773] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:00:18,773] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 18:00:18,773] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #11 test_basic_ssl_test ============== [2021-05-11 18:00:18,782] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:18,909] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:19,322] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:19,677] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:19,680] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:00:19,816] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:00:20,325] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:20,722] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:20,726] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:20,832] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:21,167] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:21,494] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:21,496] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:00:21,623] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:00:22,043] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:22,391] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:40,827] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:00:40,828] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 13.36196009093205, 'mem_free': 6831792128, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:00:40,828] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:00:40,828] - [x509tests:89] INFO - Reverting to original state - regenerating certificate and removing inbox folder [2021-05-11 18:00:40,830] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:40,972] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:41,414] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:41,780] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:41,869] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.105.151 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:00:43,261] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:43,401] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:43,817] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:44,195] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:44,196] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:00:44,198] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:44,347] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:44,782] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:45,147] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:45,150] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:00:45,302] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:00:45,724] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:46,092] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:46,093] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:00:46,208] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:00:46,233] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:46,237] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:00:46,374] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:00:46,824] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:47,208] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:47,312] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.105.153 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:00:48,385] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:00:48,531] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:00:49,001] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:49,387] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:49,388] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:00:49,391] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:00:49,535] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:00:50,034] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:50,434] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:50,436] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:00:50,574] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:00:51,034] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:51,415] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:51,416] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:00:51,539] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:00:51,562] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:51,564] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:51,661] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:51,957] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:52,277] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:52,380] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.107.160 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:00:53,925] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:54,024] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:54,320] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:54,629] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:54,630] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:00:54,633] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:54,727] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:55,011] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:55,323] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:55,325] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:00:55,421] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:00:55,724] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:56,043] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:56,044] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:00:56,104] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:00:56,119] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:00:56,122] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:00:56,257] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:00:56,684] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:00:57,031] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:00:57,155] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.107.184 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:00:59,292] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:00:59,421] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:00:59,888] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:00,246] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:00,247] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:00,249] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:00,380] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:00,789] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:01,136] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:01,138] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:01,270] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:01,736] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:02,095] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:02,096] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:01:02,204] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:01:02,222] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:02,251] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:02,390] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:02,820] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:03,181] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:03,182] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:03,184] - [remote_util:299] INFO - SSH Connecting to 127.0.0.1 with username:root, attempt#1 of 5 [2021-05-11 18:01:03,184] - [remote_util:335] INFO - SSH Connected to 127.0.0.1 as root [2021-05-11 18:01:03,252] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default [2021-05-11 18:01:03,252] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: rm -rf /tmp/newcerts8/ [2021-05-11 18:01:03,262] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,263] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: mkdir /tmp/newcerts8/ [2021-05-11 18:01:03,273] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,273] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/ca.key 1024 [2021-05-11 18:01:03,303] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n.......++++++\n......................++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,304] - [x509main:100] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n.......++++++\n......................++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,304] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -x509 -days 3650 -sha256 -key /tmp/newcerts8/ca.key -out /tmp/newcerts8/ca.pem -subj '/C=UA/O=My Company/CN=My Company Root CA' [2021-05-11 18:01:03,319] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,320] - [x509main:102] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,320] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/int.key 1024 [2021-05-11 18:01:03,363] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n................++++++\n........................................++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,364] - [x509main:104] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n................++++++\n........................................++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,364] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -key /tmp/newcerts8/int.key -out /tmp/newcerts8/int.csr -subj '/C=UA/O=My Company/CN=My Company Intermediate CA' [2021-05-11 18:01:03,380] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,380] - [x509main:106] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,380] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl x509 -req -in /tmp/newcerts8/int.csr -CA /tmp/newcerts8/ca.pem -CAkey /tmp/newcerts8/ca.key -CAcreateserial -CAserial /tmp/newcerts8/rootCA.srl -extfile ./pytests/security/v3_ca.ext -out /tmp/newcerts8/int.pem -days 365 -sha256 [2021-05-11 18:01:03,402] - [remote_util:3473] INFO - command executed with root but got an error b'Signature ok\nsubject=/C=UA/O=My Company/CN=My Company Intermediate CA\nGetting CA Private Key\n' ... [2021-05-11 18:01:03,402] - [x509main:109] INFO - Output message is b'' and error message is b'Signature ok\nsubject=/C=UA/O=My Company/CN=My Company Intermediate CA\nGetting CA Private Key\n' [ req ] default_bits = 1024 distinguished_name = req_distinguished_name req_extensions = req_ext prompt = no [ req_distinguished_name ] countryName = UA stateOrProvinceName = California localityName = Mountain View organizationName = My Company commonName = 172.23.105.151 [ req_ext ] subjectAltName = @alt_names [alt_names] IP.0 = 172.23.105.151 [2021-05-11 18:01:03,405] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/172.23.105.151.key 1024 [2021-05-11 18:01:03,427] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n......++++++\n....++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,428] - [x509main:140] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n......++++++\n....++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,428] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -key /tmp/newcerts8/172.23.105.151.key -out /tmp/newcerts8/172.23.105.151.csr -config ./pytests/security/clientconf3.conf [2021-05-11 18:01:03,444] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,444] - [x509main:142] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,444] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl x509 -req -in /tmp/newcerts8/172.23.105.151.csr -CA /tmp/newcerts8/int.pem -CAkey /tmp/newcerts8/int.key -CAcreateserial -CAserial /tmp/newcerts8/intermediateCA.srl -out /tmp/newcerts8/172.23.105.151.pem -days 365 -sha256 -extfile ./pytests/security/clientconf3.conf -extensions req_ext [2021-05-11 18:01:03,469] - [remote_util:3473] INFO - command executed with root but got an error b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.105.151\nGetting CA Private Key\n' ... [2021-05-11 18:01:03,469] - [x509main:145] INFO - Output message is b'' and error message is b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.105.151\nGetting CA Private Key\n' [2021-05-11 18:01:03,470] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: cat /tmp/newcerts8/172.23.105.151.pem /tmp/newcerts8/int.pem /tmp/newcerts8/ca.pem > /tmp/newcerts8/long_chain172.23.105.151.pem [2021-05-11 18:01:03,482] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,482] - [x509main:147] INFO - Output message is b'' and error message is b'' [ req ] default_bits = 1024 distinguished_name = req_distinguished_name req_extensions = req_ext prompt = no [ req_distinguished_name ] countryName = UA stateOrProvinceName = California localityName = Mountain View organizationName = My Company commonName = 172.23.105.153 [ req_ext ] subjectAltName = @alt_names [alt_names] IP.0 = 172.23.105.153 [2021-05-11 18:01:03,484] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/172.23.105.153.key 1024 [2021-05-11 18:01:03,519] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n........................++++++\n..............++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,520] - [x509main:140] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n........................++++++\n..............++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,520] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -key /tmp/newcerts8/172.23.105.153.key -out /tmp/newcerts8/172.23.105.153.csr -config ./pytests/security/clientconf3.conf [2021-05-11 18:01:03,535] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,536] - [x509main:142] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,536] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl x509 -req -in /tmp/newcerts8/172.23.105.153.csr -CA /tmp/newcerts8/int.pem -CAkey /tmp/newcerts8/int.key -CAcreateserial -CAserial /tmp/newcerts8/intermediateCA.srl -out /tmp/newcerts8/172.23.105.153.pem -days 365 -sha256 -extfile ./pytests/security/clientconf3.conf -extensions req_ext [2021-05-11 18:01:03,560] - [remote_util:3473] INFO - command executed with root but got an error b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.105.153\nGetting CA Private Key\n' ... [2021-05-11 18:01:03,560] - [x509main:145] INFO - Output message is b'' and error message is b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.105.153\nGetting CA Private Key\n' [2021-05-11 18:01:03,561] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: cat /tmp/newcerts8/172.23.105.153.pem /tmp/newcerts8/int.pem /tmp/newcerts8/ca.pem > /tmp/newcerts8/long_chain172.23.105.153.pem [2021-05-11 18:01:03,573] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,573] - [x509main:147] INFO - Output message is b'' and error message is b'' [ req ] default_bits = 1024 distinguished_name = req_distinguished_name req_extensions = req_ext prompt = no [ req_distinguished_name ] countryName = UA stateOrProvinceName = California localityName = Mountain View organizationName = My Company commonName = 172.23.107.160 [ req_ext ] subjectAltName = @alt_names [alt_names] IP.0 = 172.23.107.160 [2021-05-11 18:01:03,575] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/172.23.107.160.key 1024 [2021-05-11 18:01:03,607] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n..............................++++++\n.....++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,608] - [x509main:140] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n..............................++++++\n.....++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,608] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -key /tmp/newcerts8/172.23.107.160.key -out /tmp/newcerts8/172.23.107.160.csr -config ./pytests/security/clientconf3.conf [2021-05-11 18:01:03,622] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,623] - [x509main:142] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,623] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl x509 -req -in /tmp/newcerts8/172.23.107.160.csr -CA /tmp/newcerts8/int.pem -CAkey /tmp/newcerts8/int.key -CAcreateserial -CAserial /tmp/newcerts8/intermediateCA.srl -out /tmp/newcerts8/172.23.107.160.pem -days 365 -sha256 -extfile ./pytests/security/clientconf3.conf -extensions req_ext [2021-05-11 18:01:03,648] - [remote_util:3473] INFO - command executed with root but got an error b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.107.160\nGetting CA Private Key\n' ... [2021-05-11 18:01:03,650] - [x509main:145] INFO - Output message is b'' and error message is b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.107.160\nGetting CA Private Key\n' [2021-05-11 18:01:03,650] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: cat /tmp/newcerts8/172.23.107.160.pem /tmp/newcerts8/int.pem /tmp/newcerts8/ca.pem > /tmp/newcerts8/long_chain172.23.107.160.pem [2021-05-11 18:01:03,665] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,665] - [x509main:147] INFO - Output message is b'' and error message is b'' [ req ] default_bits = 1024 distinguished_name = req_distinguished_name req_extensions = req_ext prompt = no [ req_distinguished_name ] countryName = UA stateOrProvinceName = California localityName = Mountain View organizationName = My Company commonName = 172.23.107.184 [ req_ext ] subjectAltName = @alt_names [alt_names] IP.0 = 172.23.107.184 [2021-05-11 18:01:03,667] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/172.23.107.184.key 1024 [2021-05-11 18:01:03,694] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n...........++++++\n.........++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,695] - [x509main:140] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n...........++++++\n.........++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,695] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -key /tmp/newcerts8/172.23.107.184.key -out /tmp/newcerts8/172.23.107.184.csr -config ./pytests/security/clientconf3.conf [2021-05-11 18:01:03,709] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,710] - [x509main:142] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,710] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl x509 -req -in /tmp/newcerts8/172.23.107.184.csr -CA /tmp/newcerts8/int.pem -CAkey /tmp/newcerts8/int.key -CAcreateserial -CAserial /tmp/newcerts8/intermediateCA.srl -out /tmp/newcerts8/172.23.107.184.pem -days 365 -sha256 -extfile ./pytests/security/clientconf3.conf -extensions req_ext [2021-05-11 18:01:03,734] - [remote_util:3473] INFO - command executed with root but got an error b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.107.184\nGetting CA Private Key\n' ... [2021-05-11 18:01:03,734] - [x509main:145] INFO - Output message is b'' and error message is b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=172.23.107.184\nGetting CA Private Key\n' [2021-05-11 18:01:03,734] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: cat /tmp/newcerts8/172.23.107.184.pem /tmp/newcerts8/int.pem /tmp/newcerts8/ca.pem > /tmp/newcerts8/long_chain172.23.107.184.pem [2021-05-11 18:01:03,746] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,746] - [x509main:147] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,747] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: cp /tmp/newcerts8/ca.pem /tmp/newcerts8/root.crt [2021-05-11 18:01:03,756] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,756] - [x509main:150] INFO - Output message is b'' and error message is b'' [ req ] default_bits = 1024 distinguished_name = req_distinguished_name req_extensions = req_ext prompt = no [ req_distinguished_name ] countryName = UA stateOrProvinceName = California localityName = Mountain View organizationName = My Company commonName = ip_address [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = us.cbadminbucket.com URI.1 = www.cbadminbucket.com IP.0 = 172.23.107.184 [2021-05-11 18:01:03,757] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl genrsa -out /tmp/newcerts8/172.16.1.174.key 1024 [2021-05-11 18:01:03,792] - [remote_util:3473] INFO - command executed with root but got an error b'Generating RSA private key, 1024 bit long modulus\n......++++++\n................................++++++\ne is 65537 (0x10001)\n' ... [2021-05-11 18:01:03,792] - [x509main:184] INFO - Output message is b'' and error message is b'Generating RSA private key, 1024 bit long modulus\n......++++++\n................................++++++\ne is 65537 (0x10001)\n' [2021-05-11 18:01:03,792] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl req -new -key /tmp/newcerts8/172.16.1.174.key -out /tmp/newcerts8/172.16.1.174.csr -config ./pytests/security/clientconf2.conf [2021-05-11 18:01:03,809] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,809] - [x509main:186] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,809] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: openssl x509 -req -in /tmp/newcerts8/172.16.1.174.csr -CA /tmp/newcerts8/int.pem -CAkey /tmp/newcerts8/int.key -CAcreateserial -CAserial /tmp/newcerts8/intermediateCA.srl -out /tmp/newcerts8/172.16.1.174.pem -days 365 -sha256 -extfile ./pytests/security/clientconf2.conf -extensions req_ext [2021-05-11 18:01:03,832] - [remote_util:3473] INFO - command executed with root but got an error b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=ip_address\nGetting CA Private Key\n' ... [2021-05-11 18:01:03,832] - [x509main:189] INFO - Output message is b'' and error message is b'Signature ok\nsubject=/C=UA/ST=California/L=Mountain View/O=My Company/CN=ip_address\nGetting CA Private Key\n' [2021-05-11 18:01:03,832] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: cat /tmp/newcerts8/172.16.1.174.pem /tmp/newcerts8/int.pem /tmp/newcerts8/ca.pem > /tmp/newcerts8/long_chain172.16.1.174.pem [2021-05-11 18:01:03,844] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:03,845] - [x509main:191] INFO - Output message is b'' and error message is b'' [2021-05-11 18:01:03,845] - [x509tests:54] INFO - Path is ['subject.cn', 'san.dnsname', 'san.uri'] - Prefixs - ['www.cb-', 'us.', 'www.'] -- Delimeters - ['.', '.', '.'] [2021-05-11 18:01:03,845] - [x509tests:68] INFO - list of server [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 18:01:03,846] - [x509tests:69] INFO - list of server [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root, ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 18:01:03,849] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:03,985] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:04,394] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:04,762] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:04,763] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:04,766] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:04,900] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:05,311] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:05,680] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:05,681] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:05,737] - [x509main:282] INFO - Status from rest file upload command is {'cache-control': 'no-cache,no-store,must-revalidate', 'content-length': '1101', 'content-type': 'application/json', 'date': 'Wed, 12 May 2021 01:01:04 GMT', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'pragma': 'no-cache', 'server': 'Couchbase Server', 'x-content-type-options': 'nosniff', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'x-xss-protection': '1; mode=block', 'status': '200'} [2021-05-11 18:01:05,737] - [x509main:283] INFO - Content from rest file upload command is b'{"cert":{"type":"uploaded","pem":"-----BEGIN CERTIFICATE-----\\nMIICTDCCAbWgAwIBAgIJALDQbPm6itrBMA0GCSqGSIb3DQEBCwUAMD8xCzAJBgNV\\nBAYTAlVBMRMwEQYDVQQKDApNeSBDb21wYW55MRswGQYDVQQDDBJNeSBDb21wYW55\\nIFJvb3QgQ0EwHhcNMjEwNTEyMDEwMTAzWhcNMzEwNTEwMDEwMTAzWjA/MQswCQYD\\nVQQGEwJVQTETMBEGA1UECgwKTXkgQ29tcGFueTEbMBkGA1UEAwwSTXkgQ29tcGFu\\neSBSb290IENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDD8t0Fcwh+HHh2\\nvIgGf+prmG3Y6HsGsOsdiIFgkDp82i8sckOzoORto+QulBZtSYaFZwlQRff5Xx+p\\nWAcNPFK0J74ZUXzoYtzXNbystS+UoPbSj5/528U79N7Oa6N48XLYBW23UqnXdsPt\\n8WQBZyX/hVQN9EVjoLNlN+iKlymUAQIDAQABo1AwTjAdBgNVHQ4EFgQUcjTJK8FO\\nLcu15Rc/ChGGm02xIwwwHwYDVR0jBBgwFoAUcjTJK8FOLcu15Rc/ChGGm02xIwww\\nDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOBgQB/6s3sF8YyV2H/AdPBpRIs\\noiA+PWHlpJYaHHWnyy1s1+YasclnrVXA7IZwcvoDL0MfqmCYaC7pYYAeoNEwi1MD\\n2XVkVp9T/NSw/yb4kuy290j630BHl2NFeHtA9FrzDBHWDdUgZYbQ0M9H5x99VuDT\\nZy4lqFPMMoDigU9h3zL4wQ==\\n-----END CERTIFICATE-----\\n","subject":"C=UA, O=My Company, CN=My Company Root CA","expires":"2031-05-09T18:01:03.000Z"},"warnings":[{"node":"ns_1@172.23.105.151","message":"Certificate is not signed with cluster CA."}]}' [2021-05-11 18:01:05,739] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:05,874] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:06,312] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:06,674] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:06,675] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:06,677] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:06,804] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:07,205] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:07,568] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:07,633] - [remote_util:1959] INFO - Checking if the directory /opt/couchbase/var/lib/couchbase/inbox exists or not. [2021-05-11 18:01:07,635] - [remote_util:1963] INFO - Directory at /opt/couchbase/var/lib/couchbase/inbox DOES NOT exist. We will create on here [2021-05-11 18:01:07,638] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:07,772] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:08,196] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:08,557] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:08,626] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:08,760] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:09,181] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:09,541] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:09,694] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:09,838] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:10,275] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:10,636] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:10,637] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:10,751] - [x509tests:82] INFO - Into Teardown [2021-05-11 18:01:10,752] - [x509tests:89] INFO - Reverting to original state - regenerating certificate and removing inbox folder [2021-05-11 18:01:10,753] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:10,889] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:11,297] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:11,654] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:11,773] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.105.151 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:01:12,833] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:12,970] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:13,398] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:13,757] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:13,758] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:13,760] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:13,891] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:14,293] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:14,644] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:14,647] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:14,778] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:15,197] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:15,571] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:15,572] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:01:15,666] - [remote_util:1520] INFO - File chain.pem will be deleted [2021-05-11 18:01:15,693] - [remote_util:1520] INFO - File pkey.key will be deleted [2021-05-11 18:01:15,694] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:01:15,714] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:15,715] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:01:15,857] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:01:16,338] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:16,718] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:16,856] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.105.153 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:01:17,742] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:01:17,883] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:01:18,360] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:18,744] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:18,745] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:18,747] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:01:18,883] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:01:19,351] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:19,759] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:19,762] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:01:19,911] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:01:20,418] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:20,816] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:20,818] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:01:20,947] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:01:20,972] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:20,975] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:01:21,079] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:01:21,380] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:21,683] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:21,739] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.107.160 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:01:22,957] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:01:23,054] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:01:23,345] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:23,665] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:23,666] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:23,667] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:01:23,764] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:01:24,052] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:24,378] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:24,381] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:01:24,481] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:01:24,789] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:25,096] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:25,097] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:01:25,188] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:01:25,202] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:25,204] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:25,339] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:25,784] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:26,136] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:26,260] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli ssl-manage -c http://172.23.107.184 -u Administrator -p password --regenerate-cert=/tmp/abcd.pem [2021-05-11 18:01:27,540] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:27,672] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:28,094] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:28,453] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:28,454] - [x509main:235] INFO - OS type is CentOS [2021-05-11 18:01:28,457] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:28,595] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:29,019] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:29,363] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:29,366] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:29,497] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:29,953] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:30,319] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:30,320] - [x509main:256] INFO - OS type is CentOS [2021-05-11 18:01:30,433] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/inbox [2021-05-11 18:01:30,453] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:30,455] - [remote_util:299] INFO - SSH Connecting to 127.0.0.1 with username:root, attempt#1 of 5 [2021-05-11 18:01:30,455] - [remote_util:335] INFO - SSH Connected to 127.0.0.1 as root [2021-05-11 18:01:30,537] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default [2021-05-11 18:01:30,538] - [remote_util:3427] INFO - running command.raw on 127.0.0.1: rm /tmp/newcerts8/ [2021-05-11 18:01:30,549] - [remote_util:3473] INFO - command executed with root but got an error b'rm: cannot remove \xe2\x80\x98/tmp/newcerts8/\xe2\x80\x99: Is a directory\n' ... [2021-05-11 18:01:30,561] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:01:30,561] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 0.9566968781470292, 'mem_free': 6791696384, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:01:30,562] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:01:30,564] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:30,706] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:31,111] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:31,466] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:31,469] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:01:31,604] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:01:32,041] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:32,427] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:32,430] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:01:32,534] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:01:32,830] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:33,145] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:33,147] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:01:33,280] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:01:33,718] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:34,065] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:52,595] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #11 test_basic_ssl_test ============== [2021-05-11 18:01:52,619] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:01:52,631] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:01:52,639] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:01:52,649] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:01:52,655] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:52,676] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:01:52,676] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:52,681] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:01:52,688] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:01:52,689] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:52,694] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:01:52,700] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:01:52,700] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:52,705] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:01:52,711] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:01:52,711] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:52,718] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:01:52,718] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #11 test_basic_ssl_test ============== [2021-05-11 18:01:52,718] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 18:01:52,721] - [basetestcase:608] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 summary so far suite security.x509tests.x509tests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_11 ok ---------------------------------------------------------------------- Ran 1 test in 106.983s OK test_backup_restore_collection_sanity (ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_12 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest.test_backup_restore_collection_sanity,items=1000,map_data_collection=true,number_of_backups=2,load_to_collection=true,reset_services=True Test Input params: {'items': '1000', 'map_data_collection': 'true', 'number_of_backups': '2', 'load_to_collection': 'true', 'reset_services': 'True', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 12, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_12'} [2021-05-11 18:01:52,788] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:52,941] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:53,373] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:53,756] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:53,762] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:53,773] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:01:53,782] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 18:01:53,782] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #12 test_backup_restore_collection_sanity============== [2021-05-11 18:01:53,798] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:01:53,807] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:01:53,814] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:01:53,824] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:01:53,829] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:53,844] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:01:53,845] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:53,849] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:01:53,855] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:01:53,855] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:53,859] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:01:53,864] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:01:53,864] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:53,869] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:01:53,873] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:01:53,874] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:01:53,878] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:01:53,879] - [basetestcase:295] INFO - initializing cluster [2021-05-11 18:01:53,881] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:54,017] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:54,436] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:54,796] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:54,808] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:01:54,941] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:01:55,336] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:01:55,692] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:01:55,771] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:01:55,857] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:01:55,857] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service [2021-05-11 18:01:57,856] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:01:57,857] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:01:57,857] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:02:02,862] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:02:02,885] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:02,885] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:02:02,947] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:02,949] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:02:03,087] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:02:03,500] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:03,860] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:03,889] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:03,889] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:03,951] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:02:04,077] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:02:04,077] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service [2021-05-11 18:02:04,120] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:04,120] - [remote_util:338] INFO - 172.23.105.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:02:09,125] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:02:09,164] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:09,164] - [remote_util:4006] INFO - Couchbase server status: [' Process: 30460 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:02:09,164] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:09,252] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 30632 [2021-05-11 18:02:09,252] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:02:09,254] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:02:09,397] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:02:09,903] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:10,290] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:10,304] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:02:10,436] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:02:10,922] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:11,304] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:11,341] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:11,432] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:02:11,432] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service [2021-05-11 18:02:13,433] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:13,433] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:02:13,434] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:02:18,439] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:02:18,468] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:18,468] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:02:18,536] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:18,538] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:02:18,685] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:02:19,141] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:19,531] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:19,607] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:19,607] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:19,679] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:02:19,777] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:02:19,778] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service [2021-05-11 18:02:19,817] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:19,818] - [remote_util:338] INFO - 172.23.105.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:02:24,821] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:02:24,867] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:24,868] - [remote_util:4006] INFO - Couchbase server status: [' Process: 20624 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:02:24,868] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:24,950] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 20803 [2021-05-11 18:02:24,950] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:02:24,952] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:02:25,053] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:02:25,346] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:25,662] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:25,673] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:02:25,770] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:02:26,066] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:26,381] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:26,443] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:26,547] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:02:26,547] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl stop couchbase-server.service [2021-05-11 18:02:28,440] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:28,441] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:02:28,441] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:02:33,447] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:02:33,464] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:33,464] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:02:33,520] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:33,522] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:02:33,628] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:02:33,928] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:34,248] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:34,310] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:34,311] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:34,406] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:02:34,514] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:02:34,515] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl start couchbase-server.service [2021-05-11 18:02:34,556] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:34,556] - [remote_util:338] INFO - 172.23.107.160:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:02:39,561] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:02:39,595] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:39,595] - [remote_util:4006] INFO - Couchbase server status: [' Process: 9906 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:02:39,595] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:39,667] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 10048 [2021-05-11 18:02:39,667] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:02:39,669] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:02:39,810] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:02:40,247] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:40,584] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:40,596] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:02:40,737] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:02:41,157] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:41,508] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:41,576] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:41,658] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:02:41,659] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl stop couchbase-server.service [2021-05-11 18:02:43,641] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:43,641] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:02:43,641] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:02:48,645] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:02:48,667] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:48,667] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:02:48,727] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:48,730] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:02:48,867] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:02:49,319] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:02:49,690] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:02:49,762] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:02:49,762] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:49,820] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:02:49,949] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:02:49,949] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl start couchbase-server.service [2021-05-11 18:02:49,986] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:49,986] - [remote_util:338] INFO - 172.23.107.184:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:02:54,992] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:02:55,029] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:02:55,030] - [remote_util:4006] INFO - Couchbase server status: [' Process: 11015 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:02:55,030] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:02:55,115] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 11160 [2021-05-11 18:02:55,115] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:02:55,115] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 18:03:05,874] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:03:05,878] - [task:157] INFO - {'uptime': '58', 'memoryTotal': 8192827392, 'memoryFree': 6928609280, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4022, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 18:03:05,880] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:03:05,880] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 18:03:05,881] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:03:05,881] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:03:05,889] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 18:03:05,890] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:03:05,890] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:03:05,895] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 18:03:05,899] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 18:03:05,899] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 18:03:05,931] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:03:05,931] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:03:06,036] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:03:06,038] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:06,183] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:06,630] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:07,066] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:07,081] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:03:07,142] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:07,142] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:03:07,146] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:07,148] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:07,155] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:03:07,183] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:03:07,187] - [task:157] INFO - {'uptime': '43', 'memoryTotal': 8191283200, 'memoryFree': 6965575680, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:03:07,189] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:03:07,190] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 18:03:07,201] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:03:07,201] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:03:07,317] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:03:07,319] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:03:07,473] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:03:07,941] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:08,329] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:08,343] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:03:08,408] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:08,408] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:03:08,413] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:08,416] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:08,422] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:03:08,452] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:03:08,457] - [task:157] INFO - {'uptime': '27', 'memoryTotal': 7934996480, 'memoryFree': 6809391104, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:03:08,462] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:03:08,462] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:03:08,474] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:03:08,475] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:03:08,583] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:03:08,585] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:08,691] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:09,002] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:09,341] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:09,356] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:03:09,555] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:09,555] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:03:09,563] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:09,567] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:09,575] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:03:09,614] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:03:09,619] - [task:157] INFO - {'uptime': '17', 'memoryTotal': 8192131072, 'memoryFree': 6960406528, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:03:09,625] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:03:09,625] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:03:09,635] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:03:09,636] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:03:09,771] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:03:09,773] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:03:09,917] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:03:10,392] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:10,765] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:10,785] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:03:10,839] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:10,840] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:03:10,844] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:10,851] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:03:10,858] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:03:10,897] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:03:10,926] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:03:10,929] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:03:11,064] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:03:16,069] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:03:16,084] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 18:03:16,086] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:16,238] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:16,677] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:17,048] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:17,058] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:03:17,125] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:17,126] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:03:17,126] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 18:03:17,911] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3419&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:03:17,942] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:03:17,943] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:03:18,783] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:19,000] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:19,176] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:03:19,177] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:03:19,405] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:19,819] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:19,980] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:03:19,980] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:03:20,299] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:20,504] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:20,682] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:03:20,682] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:03:20,894] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:21,137] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:21,262] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:03:21,262] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:03:21,533] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:21,735] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:21,894] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:03:21,895] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:03:22,088] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:22,331] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:03:22,443] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:03:22,446] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #12 test_backup_restore_collection_sanity ============== [2021-05-11 18:03:22,456] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:22,597] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:23,001] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:23,367] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:23,371] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:03:23,515] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:03:24,031] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:24,412] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:24,415] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:24,516] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:24,812] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:25,134] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:25,137] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:03:25,277] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:03:25,717] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:26,068] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:44,029] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:03:44,029] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 23.67284734569469, 'mem_free': 6725050368, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:03:44,030] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:03:44,030] - [newupgradebasetest:55] INFO - ============== NewUpgradeBaseTest setup has started ============== [2021-05-11 18:03:44,032] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:44,170] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:44,572] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:44,928] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:44,929] - [newupgradebasetest:218] INFO - ============== NewUpgradeBaseTest setup has completed ============== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 52 100 20 100 32 10476 16762 --:--:-- --:--:-- --:--:-- 32000 [2021-05-11 18:03:44,947] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:45,073] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:45,480] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:45,835] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:45,837] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:45,942] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:46,265] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:46,580] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:46,581] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: nproc [2021-05-11 18:03:46,636] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:46,704] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:46,833] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:47,228] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:47,575] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:47,589] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:47,713] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:48,095] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:48,445] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:48,448] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:48,540] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:48,832] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:49,141] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:49,247] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:49,376] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:49,797] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:50,159] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:50,270] - [enterprise_bkrs_collection:93] INFO - *** create collection in all buckets [2021-05-11 18:03:50,271] - [enterprise_bkrs_collection:94] INFO - *** start to load items to all buckets [2021-05-11 18:03:50,272] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:03:50,401] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:03:50,797] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:51,149] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:51,151] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbworkloadgen -n 172.23.105.151:8091 -r 0.9 -i 1000 -b default -s 125 -u Administrator -p password [2021-05-11 18:03:51,692] - [remote_util:3473] INFO - command executed with root but got an error [' [####################] 100.0% (1111/estimated 1111 msgs)\x1b[A', '', 'bucket: default, msgs transferred...', ' : total | last | per sec', ' byte : 138875 | 138875 | 720901.4', 'done'] ... [2021-05-11 18:03:51,692] - [remote_util:3222] ERROR - [####################] 100.0% (1111/estimated 1111 msgs) [2021-05-11 18:03:51,693] - [remote_util:3222] ERROR - [2021-05-11 18:03:51,693] - [remote_util:3222] ERROR - bucket: default, msgs transferred... [2021-05-11 18:03:51,693] - [remote_util:3222] ERROR - : total | last | per sec [2021-05-11 18:03:51,693] - [remote_util:3222] ERROR - byte : 138875 | 138875 | 720901.4 [2021-05-11 18:03:51,693] - [remote_util:3222] ERROR - done [2021-05-11 18:03:51,693] - [enterprise_bkrs_collection:97] INFO - *** done to load items to all buckets [2021-05-11 18:03:51,724] - [rest_client:621] INFO - http://172.23.105.151:8091/pools/default/buckets/default/scopes with params: name=scope0 [2021-05-11 18:03:51,724] - [rest_client:624] INFO - Scope created default->scope0 {'uid': '1'} [2021-05-11 18:03:51,759] - [rest_client:621] INFO - http://172.23.105.151:8091/pools/default/buckets/default/scopes with params: name=scope1 [2021-05-11 18:03:51,759] - [rest_client:624] INFO - Scope created default->scope1 {'uid': '2'} [2021-05-11 18:03:51,817] - [rest_client:644] INFO - http://172.23.105.151:8091/pools/default/buckets/default/scopes/scope1/collections with params: name=mycollection_scope1_0 [2021-05-11 18:03:51,818] - [rest_client:655] INFO - Collection created default->scope1->mycollection_scope1_0 manifest:{'uid': '3'} [2021-05-11 18:03:51,857] - [rest_client:644] INFO - http://172.23.105.151:8091/pools/default/buckets/default/scopes/scope0/collections with params: name=mycollection_scope0_0 [2021-05-11 18:03:51,857] - [rest_client:655] INFO - Collection created default->scope0->mycollection_scope0_0 manifest:{'uid': '4'} [2021-05-11 18:03:51,864] - [enterprise_bkrs_collection:110] INFO - scopes in backup cluster: ['scope1', 'scope0', '_default'] [2021-05-11 18:03:51,865] - [enterprise_bkrs_collection:115] INFO - get scope id of scope: scope1 [2021-05-11 18:03:52,078] - [enterprise_bkrs_collection:115] INFO - get scope id of scope: scope0 [2021-05-11 18:03:52,342] - [enterprise_bkrs_collection:117] INFO - scope id in backup cluster: ['0x9', '0x8'] [2021-05-11 18:03:53,283] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:53,379] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:53,666] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:53,974] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:53,975] - [enterprise_bkrs_collection_base:736] INFO - Remove any old dir before create new one [2021-05-11 18:03:53,975] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /tmp/entbackup_172.23.105.151 [2021-05-11 18:03:54,029] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:54,030] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr config --archive /tmp/entbackup_172.23.105.151 --repo backup [2021-05-11 18:03:54,112] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:54,112] - [remote_util:3260] INFO - Backup repository `backup` created successfully in archive `/tmp/entbackup_172.23.105.151` [2021-05-11 18:03:54,114] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:54,207] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:54,486] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:54,790] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:54,791] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -R /tmp/entbackup_172.23.105.151/backup [2021-05-11 18:03:54,846] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:03:54,850] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:03:54,940] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:03:55,227] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:03:55,533] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:03:55,596] - [enterprise_bkrs_collection_base:787] INFO - Expected Backup Meta data json and Actual Backup Meta data json are equal Backup create validation success. [2021-05-11 18:03:55,597] - [enterprise_bkrs_collection:148] INFO - *** start to update items in all buckets [2021-05-11 18:03:55,597] - [basetestcase:662] INFO - sleep for 10 secs. wait for scopes and collections created ... [2021-05-11 18:04:05,609] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:04:05,742] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:04:06,122] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:06,471] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:06,472] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbworkloadgen -n 172.23.105.151:8091 -r 0.1 -i 1000 -b default -s 125 -c 0x9 -u Administrator -p password [2021-05-11 18:04:07,436] - [remote_util:3473] INFO - command executed with root but got an error [' [######## ] 40.0% (4000/estimated 9991 msgs)\x1b[A', ' [################## ] 90.1% (9000/estimated 9991 msgs)\x1b[A', ' [####################] 100.0% (9991/estimated 9991 msgs)\x1b[A', '', 'bucket: default, msgs transferred...', ' : total | last | per sec', ' byte : 1248875 | 1248875 | 1998434.8', 'done'] ... [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - [######## ] 40.0% (4000/estimated 9991 msgs) [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - [################## ] 90.1% (9000/estimated 9991 msgs) [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - [####################] 100.0% (9991/estimated 9991 msgs) [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - bucket: default, msgs transferred... [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - : total | last | per sec [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - byte : 1248875 | 1248875 | 1998434.8 [2021-05-11 18:04:07,437] - [remote_util:3222] ERROR - done [2021-05-11 18:04:07,438] - [enterprise_bkrs_collection:169] INFO - *** done update items in all buckets [2021-05-11 18:04:07,438] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 18:04:17,448] - [enterprise_bkrs_collection:171] INFO - *** start to validate backup cluster [2021-05-11 18:04:17,454] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:04:17,466] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:04:17,468] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:17,565] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:17,850] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:18,165] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:18,166] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --no-progress-bar [2021-05-11 18:04:21,952] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:21,952] - [enterprise_bkrs_collection_base:1841] INFO - Found 'Backup completed successfully in CLI output [2021-05-11 18:04:21,952] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -tr /tmp/entbackup_172.23.105.151/backup | tail -1 [2021-05-11 18:04:22,006] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:22,007] - [enterprise_bkrs_collection_base:881] INFO - Finished taking backup with args: backup --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --no-progress-bar [2021-05-11 18:04:22,008] - [enterprise_bkrs_collection_base:1841] INFO - Found 'Backup completed successfully in CLI output [2021-05-11 18:04:22,010] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:22,106] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:22,390] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:22,741] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:22,742] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr info --all -j --archive /tmp/entbackup_172.23.105.151 [2021-05-11 18:04:22,837] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:22,840] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:22,933] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:23,211] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:23,522] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:23,523] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -lR /tmp/entbackup_172.23.105.151/backup [2021-05-11 18:04:23,646] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:23,690] - [enterprise_bkrs_collection_base:904] INFO - Expected Directory Structure and Actual Directory Structure are equal [2021-05-11 18:04:23,730] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:04:24,308] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:04:24,436] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:04:24,827] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:25,190] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:25,192] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 18:04:26,035] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 18:04:26,133] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:04:26,264] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:04:26,657] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:27,014] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:27,015] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 18:04:28,085] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 18:04:28,162] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:28,265] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:28,565] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:28,873] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:28,882] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: cat /tmp/entbackup_172.23.105.151/backup/2021-05-11T18_04_18.279671721-07_00/default-a597d2769c60075ad0f375dd0cb582ca/range.json [2021-05-11 18:04:28,960] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:28,970] - [enterprise_bkrs_collection:148] INFO - *** start to update items in all buckets [2021-05-11 18:04:28,970] - [basetestcase:662] INFO - sleep for 10 secs. wait for scopes and collections created ... [2021-05-11 18:04:38,974] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:04:39,116] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:04:39,508] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:39,867] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:39,868] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbworkloadgen -n 172.23.105.151:8091 -r 0.1 -i 1000 -b default -s 125 -c 0x9 -u Administrator -p password [2021-05-11 18:04:40,825] - [remote_util:3473] INFO - command executed with root but got an error [' [######## ] 40.0% (4000/estimated 9991 msgs)\x1b[A', ' [################## ] 90.1% (9000/estimated 9991 msgs)\x1b[A', ' [####################] 100.0% (9991/estimated 9991 msgs)\x1b[A', '', 'bucket: default, msgs transferred...', ' : total | last | per sec', ' byte : 1248875 | 1248875 | 2071547.0', 'done'] ... [2021-05-11 18:04:40,825] - [remote_util:3222] ERROR - [######## ] 40.0% (4000/estimated 9991 msgs) [2021-05-11 18:04:40,825] - [remote_util:3222] ERROR - [################## ] 90.1% (9000/estimated 9991 msgs) [2021-05-11 18:04:40,825] - [remote_util:3222] ERROR - [####################] 100.0% (9991/estimated 9991 msgs) [2021-05-11 18:04:40,825] - [remote_util:3222] ERROR - [2021-05-11 18:04:40,826] - [remote_util:3222] ERROR - bucket: default, msgs transferred... [2021-05-11 18:04:40,826] - [remote_util:3222] ERROR - : total | last | per sec [2021-05-11 18:04:40,826] - [remote_util:3222] ERROR - byte : 1248875 | 1248875 | 2071547.0 [2021-05-11 18:04:40,826] - [remote_util:3222] ERROR - done [2021-05-11 18:04:40,826] - [enterprise_bkrs_collection:169] INFO - *** done update items in all buckets [2021-05-11 18:04:40,826] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 18:04:50,836] - [enterprise_bkrs_collection:171] INFO - *** start to validate backup cluster [2021-05-11 18:04:50,846] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:04:50,865] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:04:50,867] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:50,968] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:51,264] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:51,571] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:51,572] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr backup --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --no-progress-bar [2021-05-11 18:04:56,026] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:56,027] - [enterprise_bkrs_collection_base:1841] INFO - Found 'Backup completed successfully in CLI output [2021-05-11 18:04:56,027] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -tr /tmp/entbackup_172.23.105.151/backup | tail -1 [2021-05-11 18:04:56,083] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:56,084] - [enterprise_bkrs_collection_base:881] INFO - Finished taking backup with args: backup --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --no-progress-bar [2021-05-11 18:04:56,084] - [enterprise_bkrs_collection_base:1841] INFO - Found 'Backup completed successfully in CLI output [2021-05-11 18:04:56,087] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:56,226] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:56,510] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:56,820] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:56,822] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr info --all -j --archive /tmp/entbackup_172.23.105.151 [2021-05-11 18:04:56,938] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:56,942] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:04:57,045] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:04:57,338] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:57,642] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:57,643] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: ls -lR /tmp/entbackup_172.23.105.151/backup [2021-05-11 18:04:57,831] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:04:57,920] - [enterprise_bkrs_collection_base:904] INFO - Expected Directory Structure and Actual Directory Structure are equal [2021-05-11 18:04:57,972] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:04:58,508] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:04:58,641] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:04:59,029] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:04:59,391] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:04:59,392] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 18:05:00,249] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 18:05:00,348] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:05:00,486] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:05:00,869] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:05:01,239] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:05:01,240] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 18:05:02,087] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 18:05:02,166] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:05:02,271] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:05:02,561] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:05:02,866] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:05:02,874] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: cat /tmp/entbackup_172.23.105.151/backup/2021-05-11T18_04_51.684452222-07_00/default-a597d2769c60075ad0f375dd0cb582ca/range.json [2021-05-11 18:05:02,952] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:05:02,962] - [enterprise_bkrs_collection:179] INFO - *** start to restore cluster [2021-05-11 18:05:02,962] - [enterprise_bkrs_collection:183] INFO - *** start to reset cluster [2021-05-11 18:05:03,019] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 18:05:03,537] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:05:03,541] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:05:03,541] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 18:05:03,548] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:05:03,567] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:05:03,572] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:05:03,698] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:05:04,107] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:05:04,480] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:05:04,491] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:05:04,553] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:05:04,553] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:05:04,560] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 18:05:04,566] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 18:05:05,569] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:07,572] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:09,576] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:11,580] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:13,588] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 18:05:21,606] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:05:21,606] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:05:21,606] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:05:21,611] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:05:21,611] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:05:21,611] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:05:21,616] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 18:05:21,617] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 18:05:21,622] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 18:05:21,623] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 18:05:21,650] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:05:21,651] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:05:21,738] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:05:21,738] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:05:21,745] - [enterprise_bkrs_collection:197] INFO - Done reset cluster [2021-05-11 18:05:21,745] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 18:05:31,762] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:05:31,775] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:05:31,776] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:05:31,870] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:05:36,875] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:05:36,891] - [enterprise_bkrs_collection:205] INFO - *** start restore validation [2021-05-11 18:05:36,899] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:05:36,908] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:05:37,037] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:05:37,425] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:05:37,767] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:05:37,776] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:05:37,837] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:05:37,838] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:05:37,838] - [enterprise_bkrs_collection_base:1022] INFO - Services in restore cluster: ['backup', 'fts', 'index', 'kv', 'n1ql'] [2021-05-11 18:05:37,851] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:05:37,851] - [enterprise_bkrs_collection_base:1055] INFO - Creating bucket default in restore host 172.23.105.151 [2021-05-11 18:05:37,862] - [rest_client:41] INFO - -->is_ns_server_running? bk service in reset restore: [['backup', 'fts', 'index', 'kv', 'n1ql']] service: ['backup', 'fts', 'index', 'kv', 'n1ql'] [2021-05-11 18:05:37,895] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 18:05:37,903] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 18:05:38,905] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:40,909] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:42,912] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:44,917] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:05:47,024] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 18:05:54,031] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:05:54,041] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:05:54,173] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:05:54,582] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:05:54,949] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:05:54,956] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:05:54,956] - [remote_util:5217] INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2021-05-11 18:05:55,966] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:05:55,966] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:05:55,966] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:05:55,971] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:05:55,971] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:05:55,971] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:05:55,976] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 18:05:55,976] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 18:05:55,981] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 18:05:55,982] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 18:05:56,009] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:05:56,010] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:05:56,109] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:05:56,117] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:05:56,246] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:05:56,666] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:05:57,059] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:05:57,076] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:05:57,138] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:05:57,139] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:05:57,139] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:05:57,146] - [enterprise_bkrs_collection_base:1878] INFO - Done reset node with storage mode [2021-05-11 18:05:58,158] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:05:58,158] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:05:58,158] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:05:58,165] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:05:58,166] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:05:58,166] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:05:58,173] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 18:05:58,173] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 18:05:58,178] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 18:05:58,179] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 18:05:58,181] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 18:05:58,182] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 18:05:58,182] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:05:58,182] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:05:58,273] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:05:58,273] - [basetestcase:662] INFO - sleep for 12 secs. time needs for index service is up (MB-39859) ... [2021-05-11 18:06:10,285] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 18:06:20,308] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:06:20,309] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:06:39,211] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:06:39,242] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:06:49,249] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:06:49,257] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.014479637145996 sec [2021-05-11 18:06:49,259] - [enterprise_bkrs_collection_base:1075] INFO - replica in bucket default is 1 [2021-05-11 18:06:49,266] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3571&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:06:49,297] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:06:49,902] - [basetestcase:662] INFO - sleep for 5 secs. wait for bucket is up ... [2021-05-11 18:06:54,938] - [basetestcase:662] INFO - sleep for 15 secs. wait for index service ready ... [2021-05-11 18:07:09,955] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:07:10,058] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:07:10,359] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:07:10,678] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:07:10,689] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/node/controller/rename body: username=Administrator&password=password&hostname=172.23.105.151 headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["Renaming is disallowed for nodes that are already part of a cluster"]' auth: Administrator:password [2021-05-11 18:07:10,690] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr restore --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --start 2021-05-11T18_04_51.684452222-07_00 --end 2021-05-11T18_04_51.684452222-07_00 --no-progress-bar --map-data default.scope1=default.scope1,default.scope0=default.scope0 [2021-05-11 18:07:12,950] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:07:12,950] - [remote_util:3260] INFO - | Transfer [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - | -------- [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - | Status | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - | Succeeded | 61.98KiB | Tue, 11 May 2021 18:07:10 -0700/s | Tue, 11 May 2021 18:07:12 -0700 | 2.162s | [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - | Bucket [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - | ------ [2021-05-11 18:07:12,951] - [remote_util:3260] INFO - | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - | default | Succeeded | 123.96KiB | 123.96KiB/s | Tue, 11 May 2021 18:07:11 -0700 | Tue, 11 May 2021 18:07:12 -0700 | 1.717s | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - | Mutations | Deletions | Expirations | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - | --------- | --------- | ----------- | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - | 1000 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - [2021-05-11 18:07:12,952] - [remote_util:3260] INFO - Restore completed successfully [2021-05-11 18:07:12,962] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:07:13,056] - [enterprise_bkrs_collection_base:642] INFO - Scope scope0 restored [2021-05-11 18:07:13,056] - [enterprise_bkrs_collection_base:642] INFO - Scope scope1 restored [2021-05-11 18:07:13,057] - [enterprise_bkrs_collection_base:642] INFO - Scope _default restored [2021-05-11 18:07:13,064] - [enterprise_bkrs_collection_base:670] INFO - Collection mycollection_scope0_0 restored [2021-05-11 18:07:13,069] - [enterprise_bkrs_collection_base:670] INFO - Collection mycollection_scope1_0 restored [2021-05-11 18:07:13,070] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:07:13,169] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:07:13,475] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:07:13,798] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:07:13,808] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:07:13,809] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: grep 'Restore completed successfully' /tmp/entbackup_172.23.105.151/logs/backup-*.log [2021-05-11 18:07:13,852] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:07:13,853] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: grep 'Transfer failed' /tmp/entbackup_172.23.105.151/logs/backup-*.log [2021-05-11 18:07:13,906] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:07:13,907] - [enterprise_bkrs_collection_base:1295] INFO - Finished restoring backup [2021-05-11 18:07:13,936] - [enterprise_bkrs_collection_base:1296] INFO - Get current vseqno on node 172.23.105.151 [2021-05-11 18:07:13,944] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:07:13,958] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:07:13,959] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:07:14,056] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:07:19,061] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:07:19,147] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:07:19,701] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:07:20,274] - [enterprise_bkrs_collection_base:1305] INFO - *** Start to validate the restore [2021-05-11 18:07:20,274] - [backup_restore_validations:91] INFO - backup start: 2 [2021-05-11 18:07:20,274] - [backup_restore_validations:92] INFO - backup end: 2 [2021-05-11 18:07:20,284] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:07:20,418] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:07:20,825] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:07:21,184] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:07:21,185] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 18:07:52,471] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 18:07:52,541] - [backup_restore_validations:131] INFO - Compare backup data in bucket default [2021-05-11 18:07:52,541] - [enterprise_bkrs_collection_base:1326] INFO - End Verification for vbucket sequence numbers comparison Expected default Data and Actual default Data are equal Data validation success [2021-05-11 18:07:52,542] - [enterprise_bkrs_collection:183] INFO - *** start to reset cluster [2021-05-11 18:07:52,598] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 18:07:54,004] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:07:54,008] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:07:54,008] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 18:07:54,017] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:07:54,023] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:07:54,036] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:07:54,039] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:07:54,095] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:07:54,110] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:08:04,124] - [rest_client:1894] INFO - rebalance percentage : 53.00 % [2021-05-11 18:08:14,138] - [rest_client:1894] INFO - rebalance percentage : 53.00 % [2021-05-11 18:08:34,161] - [rest_client:1833] INFO - rebalance progress took 40.07 seconds [2021-05-11 18:08:34,161] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:08:44,184] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 18:08:44,190] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:08:44,199] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:08:44,207] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:08:44,345] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:08:44,752] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:08:45,114] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:08:45,129] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:08:45,193] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:08:45,194] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:08:45,202] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 18:08:45,206] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 18:08:46,210] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:08:48,213] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:08:50,216] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:08:52,220] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:08:54,290] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 18:09:02,308] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:09:02,309] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:09:02,309] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:09:02,314] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:09:02,314] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:09:02,315] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:09:02,318] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 18:09:02,319] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 18:09:02,324] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 18:09:02,324] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 18:09:02,351] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:09:02,352] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:09:02,450] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:09:02,451] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:09:02,457] - [enterprise_bkrs_collection:197] INFO - Done reset cluster [2021-05-11 18:09:02,457] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 18:09:12,478] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:09:12,497] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:09:12,497] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:09:12,598] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:09:17,604] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:09:17,624] - [enterprise_bkrs_collection:205] INFO - *** start restore validation [2021-05-11 18:09:17,637] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:09:17,653] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:09:17,799] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:09:18,190] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:09:18,562] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:09:18,574] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:09:18,643] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:09:18,643] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:09:18,644] - [enterprise_bkrs_collection_base:1022] INFO - Services in restore cluster: ['backup', 'fts', 'index', 'kv', 'n1ql'] [2021-05-11 18:09:18,662] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:09:18,663] - [enterprise_bkrs_collection_base:1055] INFO - Creating bucket default in restore host 172.23.105.151 [2021-05-11 18:09:18,676] - [rest_client:41] INFO - -->is_ns_server_running? bk service in reset restore: [['backup', 'fts', 'index', 'kv', 'n1ql']] service: ['backup', 'fts', 'index', 'kv', 'n1ql'] [2021-05-11 18:09:18,710] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 18:09:18,714] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 18:09:19,716] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:09:21,720] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:09:23,725] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:09:25,730] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:09:27,743] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 18:09:34,751] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:09:34,759] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:09:34,905] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:09:35,363] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:09:35,730] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:09:35,739] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:09:35,740] - [remote_util:5217] INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2021-05-11 18:09:36,751] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:09:36,751] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:09:36,751] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:09:36,757] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:09:36,757] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:09:36,757] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:09:36,761] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 18:09:36,761] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 18:09:36,765] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 18:09:36,766] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 18:09:36,791] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:09:36,792] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:09:36,906] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:09:36,914] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:09:37,043] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:09:37,487] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:09:37,857] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:09:37,867] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:09:37,936] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:09:37,937] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:09:37,937] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:09:37,945] - [enterprise_bkrs_collection_base:1878] INFO - Done reset node with storage mode [2021-05-11 18:09:38,956] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:09:38,957] - [rest_client:1083] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:09:38,957] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:09:38,962] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:09:38,962] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:09:38,963] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:09:38,968] - [rest_client:1099] INFO - quota for kv: 4465 MB [2021-05-11 18:09:38,968] - [rest_client:1147] INFO - pools/default params : memoryQuota=4465 [2021-05-11 18:09:38,972] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['backup', 'fts', 'index', 'kv', 'n1ql']) [2021-05-11 18:09:38,972] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql [2021-05-11 18:09:38,975] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=backup%2Cfts%2Cindex%2Ckv%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 18:09:38,976] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 18:09:38,976] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:09:38,976] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:09:39,059] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:09:39,059] - [basetestcase:662] INFO - sleep for 12 secs. time needs for index service is up (MB-39859) ... [2021-05-11 18:09:51,071] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 18:10:01,090] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:10:01,090] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:10:20,482] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:10:20,509] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:10:30,514] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:10:30,521] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.011767625808716 sec [2021-05-11 18:10:30,524] - [enterprise_bkrs_collection_base:1075] INFO - replica in bucket default is 1 [2021-05-11 18:10:30,532] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3571&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:10:30,564] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:10:31,157] - [basetestcase:662] INFO - sleep for 5 secs. wait for bucket is up ... [2021-05-11 18:10:36,194] - [basetestcase:662] INFO - sleep for 15 secs. wait for index service ready ... [2021-05-11 18:10:51,206] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:10:51,311] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:10:51,616] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:10:51,943] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:10:51,957] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/node/controller/rename body: username=Administrator&password=password&hostname=172.23.105.151 headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["Renaming is disallowed for nodes that are already part of a cluster"]' auth: Administrator:password [2021-05-11 18:10:51,958] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin//cbbackupmgr restore --archive /tmp/entbackup_172.23.105.151 --repo backup --cluster http://172.23.105.151:8091 --username Administrator --password password --start 2021-05-11T18_04_18.279671721-07_00 --end 2021-05-11T18_04_51.684452222-07_00 --no-progress-bar --map-data default.scope1=default.scope1,default.scope0=default.scope0 [2021-05-11 18:10:55,852] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:10:55,852] - [remote_util:3260] INFO - | Transfer [2021-05-11 18:10:55,852] - [remote_util:3260] INFO - | -------- [2021-05-11 18:10:55,853] - [remote_util:3260] INFO - | Status | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 18:10:55,853] - [remote_util:3260] INFO - | Succeeded | 61.98KiB | Tue, 11 May 2021 18:10:52 -0700/s | Tue, 11 May 2021 18:10:54 -0700 | 2.181s | [2021-05-11 18:10:55,854] - [remote_util:3260] INFO - [2021-05-11 18:10:55,854] - [remote_util:3260] INFO - | Bucket [2021-05-11 18:10:55,854] - [remote_util:3260] INFO - | ------ [2021-05-11 18:10:55,854] - [remote_util:3260] INFO - | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 18:10:55,854] - [remote_util:3260] INFO - | default | Succeeded | 123.96KiB | 123.96KiB/s | Tue, 11 May 2021 18:10:52 -0700 | Tue, 11 May 2021 18:10:54 -0700 | 1.674s | [2021-05-11 18:10:55,854] - [remote_util:3260] INFO - | [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | Mutations | Deletions | Expirations | [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | --------- | --------- | ----------- | [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | 1000 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | Transfer [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | -------- [2021-05-11 18:10:55,855] - [remote_util:3260] INFO - | Status | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | Succeeded | 41.32KiB | Tue, 11 May 2021 18:10:52 -0700/s | Tue, 11 May 2021 18:10:55 -0700 | 3.783s | [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | Bucket [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | ------ [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | default | Succeeded | 123.96KiB | 123.96KiB/s | Tue, 11 May 2021 18:10:54 -0700 | Tue, 11 May 2021 18:10:55 -0700 | 1.385s | [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | Mutations | Deletions | Expirations | [2021-05-11 18:10:55,856] - [remote_util:3260] INFO - | --------- | --------- | ----------- | [2021-05-11 18:10:55,857] - [remote_util:3260] INFO - | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | [2021-05-11 18:10:55,857] - [remote_util:3260] INFO - | 1000 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | [2021-05-11 18:10:55,857] - [remote_util:3260] INFO - [2021-05-11 18:10:55,857] - [remote_util:3260] INFO - Restore completed successfully [2021-05-11 18:10:55,868] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:10:55,917] - [enterprise_bkrs_collection_base:642] INFO - Scope scope0 restored [2021-05-11 18:10:55,917] - [enterprise_bkrs_collection_base:642] INFO - Scope scope1 restored [2021-05-11 18:10:55,917] - [enterprise_bkrs_collection_base:642] INFO - Scope _default restored [2021-05-11 18:10:55,924] - [enterprise_bkrs_collection_base:670] INFO - Collection mycollection_scope0_0 restored [2021-05-11 18:10:55,929] - [enterprise_bkrs_collection_base:670] INFO - Collection mycollection_scope1_0 restored [2021-05-11 18:10:55,930] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:10:56,027] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:10:56,302] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:10:56,611] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:10:56,624] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:10:56,624] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: grep 'Restore completed successfully' /tmp/entbackup_172.23.105.151/logs/backup-*.log [2021-05-11 18:10:56,668] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:10:56,669] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: grep 'Transfer failed' /tmp/entbackup_172.23.105.151/logs/backup-*.log [2021-05-11 18:10:56,723] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:10:56,724] - [enterprise_bkrs_collection_base:1295] INFO - Finished restoring backup [2021-05-11 18:10:56,762] - [enterprise_bkrs_collection_base:1296] INFO - Get current vseqno on node 172.23.105.151 [2021-05-11 18:10:56,773] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:10:56,793] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:10:56,794] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:10:56,904] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:11:01,910] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:11:02,009] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:11:02,480] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:11:03,042] - [enterprise_bkrs_collection_base:1305] INFO - *** Start to validate the restore [2021-05-11 18:11:03,043] - [backup_restore_validations:91] INFO - backup start: 1 [2021-05-11 18:11:03,043] - [backup_restore_validations:92] INFO - backup end: 2 [2021-05-11 18:11:03,052] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:11:03,213] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:11:03,631] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:11:04,040] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:11:04,040] - [remote_util:4397] INFO - Run cbtransfer to get data map [2021-05-11 18:11:35,339] - [remote_util:4436] INFO - done execute cbtransfer [2021-05-11 18:11:35,409] - [backup_restore_validations:131] INFO - Compare backup data in bucket default [2021-05-11 18:11:35,409] - [enterprise_bkrs_collection_base:1326] INFO - End Verification for vbucket sequence numbers comparison Expected default Data and Actual default Data are equal Data validation success [2021-05-11 18:11:35,424] - [enterprise_bkrs_collection_base:670] INFO - Collection mycollection_scope0_0 restored [2021-05-11 18:11:35,432] - [enterprise_bkrs_collection_base:670] INFO - Collection mycollection_scope1_0 restored [2021-05-11 18:11:35,432] - [newupgradebasetest:230] INFO - ============== NewUpgradeBaseTest tearDown has started ============== [2021-05-11 18:11:35,453] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:11:35,453] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.01816347124117, 'mem_free': 6684999680, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:11:35,453] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.769385699899295, 'mem_free': 6766096384, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:11:35,453] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:11:35,455] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:11:35,587] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:11:36,008] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:11:36,367] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:11:36,370] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:11:36,510] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:11:36,978] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:11:37,344] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:11:51,835] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #12 test_backup_restore_collection_sanity ============== [2021-05-11 18:11:51,881] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 18:11:53,376] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:11:53,380] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:11:53,380] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 18:11:53,388] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:11:53,393] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:11:53,405] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:11:53,407] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:11:53,456] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:11:53,460] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:12:03,480] - [rest_client:1894] INFO - rebalance percentage : 53.00 % [2021-05-11 18:12:13,494] - [rest_client:1894] INFO - rebalance percentage : 53.00 % [2021-05-11 18:12:33,523] - [rest_client:1833] INFO - rebalance progress took 40.07 seconds [2021-05-11 18:12:33,523] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:12:43,543] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 18:12:43,549] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:12:43,550] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:12:43,554] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:12:43,559] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:12:43,559] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:12:43,563] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:12:43,563] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #12 test_backup_restore_collection_sanity ============== [2021-05-11 18:12:43,563] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 18:12:43,564] - [basetestcase:608] INFO - closing all memcached connections [2021-05-11 18:12:43,566] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:12:43,701] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:12:44,114] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:12:44,463] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:12:44,475] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:12:44,597] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:12:44,975] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:12:45,320] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:12:45,344] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:12:45,416] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:12:45,416] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service [2021-05-11 18:12:47,249] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:12:47,250] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:12:47,250] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:12:52,255] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:12:52,281] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:12:52,282] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:12:52,344] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:12:52,346] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:12:52,471] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:12:52,857] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:12:53,206] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:12:53,274] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:12:53,274] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:12:53,366] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:12:53,480] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:12:53,480] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service [2021-05-11 18:12:53,514] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:12:53,515] - [remote_util:338] INFO - 172.23.105.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:12:58,520] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:12:58,557] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:12:58,558] - [remote_util:4006] INFO - Couchbase server status: [' Process: 6851 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:12:58,558] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:12:58,637] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 7023 [2021-05-11 18:12:58,637] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:12:58,640] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:12:58,816] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:12:59,315] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:12:59,736] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:12:59,749] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:12:59,884] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:13:00,308] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:00,684] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:00,715] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:13:00,807] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:13:00,807] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service [2021-05-11 18:13:02,635] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:02,636] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:13:02,636] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:13:07,641] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:13:07,672] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:07,672] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:13:07,735] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:07,737] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:13:07,873] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:13:08,311] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:08,672] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:08,743] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:13:08,743] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:13:08,805] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:13:08,930] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:13:08,931] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service [2021-05-11 18:13:08,965] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:08,965] - [remote_util:338] INFO - 172.23.105.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:13:13,971] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:13:14,009] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:14,009] - [remote_util:4006] INFO - Couchbase server status: [' Process: 26643 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:13:14,009] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:13:14,099] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 26865 [2021-05-11 18:13:14,099] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:13:14,100] - [basetestcase:662] INFO - sleep for 10 secs. ... Cluster instance shutdown with force [2021-05-11 18:13:24,122] - [basetestcase:662] INFO - sleep for 20 secs. sleep 20 seconds before run next test ... [2021-05-11 18:13:44,143] - [newupgradebasetest:269] INFO - ============== NewUpgradeBaseTest tearDown has completed ============== [2021-05-11 18:13:44,145] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:13:44,244] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:13:44,572] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:44,887] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:44,888] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /tmp/entbackup_172.23.105.151 [2021-05-11 18:13:45,272] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:45,272] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /cbqe3043/entbackup [2021-05-11 18:13:45,326] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:45,341] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:13:45,357] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:13:45,362] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:45,380] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:45,388] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:45,399] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:13:45,406] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:13:45,410] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:45,431] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:45,438] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:45,442] - [enterprise_bkrs_collection_base:409] INFO - delete dir /tmp/backuprestore172.23.105.151 summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 summary so far suite security.x509tests.x509tests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_12 ok ---------------------------------------------------------------------- Ran 1 test in 712.660s OK test_export_and_import_back (clitest.importexporttest.ImportExportTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_13 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t clitest.importexporttest.ImportExportTests.test_export_and_import_back,imex_type=json,format_type=list,import_back=True,nodes_init=2 Test Input params: {'imex_type': 'json', 'format_type': 'list', 'import_back': 'True', 'nodes_init': '2', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 13, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_13'} [2021-05-11 18:13:45,499] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:13:45,631] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:13:46,030] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:46,381] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:46,387] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:46,399] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:13:46,405] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:13:46,407] - [rest_client:2475] INFO - Node versions in cluster [] [2021-05-11 18:13:46,407] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #13 test_export_and_import_back============== [2021-05-11 18:13:46,413] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:13:46,425] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:13:46,432] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:13:46,441] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:13:46,448] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:13:46,454] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:46,467] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:13:46,468] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:46,471] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:13:46,476] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:13:46,476] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:46,480] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:13:46,485] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:13:46,485] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:46,489] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:13:46,495] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:13:46,495] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:13:46,500] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:13:46,500] - [basetestcase:295] INFO - initializing cluster [2021-05-11 18:13:47,507] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:13:47,512] - [task:157] INFO - {'uptime': '51', 'memoryTotal': 8192827392, 'memoryFree': 6919680000, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4022, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 18:13:47,516] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:13:47,517] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 18:13:47,517] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:13:47,517] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:13:47,525] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 18:13:47,525] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:13:47,525] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:13:47,529] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 18:13:47,536] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 18:13:47,536] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 18:13:47,562] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:13:47,562] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:13:47,651] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:13:47,653] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:13:47,788] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:13:48,244] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:48,632] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:48,644] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:13:48,704] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:48,705] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:13:48,709] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:48,712] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:48,718] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:13:48,746] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:13:48,753] - [task:157] INFO - {'uptime': '37', 'memoryTotal': 8191283200, 'memoryFree': 6960455680, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:13:48,758] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:13:48,758] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 18:13:48,769] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:13:48,769] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:13:48,865] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:13:48,867] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:13:49,011] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:13:49,464] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:49,861] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:49,875] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:13:49,943] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:49,944] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:13:49,951] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:49,954] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:49,965] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:13:49,998] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:13:50,008] - [task:157] INFO - {'uptime': '671', 'memoryTotal': 7934996480, 'memoryFree': 6826749952, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4055, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:13:50,015] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:13:50,025] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:13:50,025] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:13:50,113] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:13:50,117] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:13:50,225] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:13:50,510] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:50,823] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:50,838] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:13:51,042] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:51,042] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:13:51,046] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:51,049] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:51,054] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:13:51,071] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:13:51,079] - [task:157] INFO - {'uptime': '655', 'memoryTotal': 8192131072, 'memoryFree': 6959579136, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:13:51,088] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:13:51,095] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:13:51,095] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:13:51,185] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:13:51,187] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:13:51,323] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:13:51,724] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:13:52,092] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:13:52,105] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:13:52,167] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:13:52,167] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:13:52,170] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:52,174] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:13:52,179] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:13:52,207] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:13:52,230] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:13:52,231] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:13:52,355] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:13:57,357] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:13:57,370] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 18:13:58,201] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 18:13:58,202] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 18:14:08,219] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:14:08,220] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:14:27,167] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:14:27,198] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:14:37,213] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:14:37,218] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.01978063583374 sec [2021-05-11 18:14:37,220] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:14:37,350] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:14:37,739] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:14:38,089] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:14:38,100] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:14:38,165] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:14:38,165] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:14:38,166] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 18:14:38,241] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3419&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:14:38,272] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:14:38,272] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:14:39,111] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:39,248] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:39,483] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:39,701] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:39,856] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:14:39,856] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:14:40,229] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:40,418] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:40,727] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:40,882] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:41,077] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:14:41,077] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:14:41,385] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:41,532] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:41,811] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:41,972] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:42,109] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:14:42,109] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:14:42,375] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:42,521] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:42,800] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:42,941] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:43,133] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:14:43,133] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:14:43,342] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:43,486] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:43,759] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:43,893] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:44,074] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:14:44,074] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:14:44,274] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:44,466] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:44,679] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:14:44,815] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:14:44,992] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:14:44,996] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #13 test_export_and_import_back ============== [2021-05-11 18:14:45,005] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:14:45,129] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:14:45,516] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:14:45,869] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:14:45,872] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:14:46,013] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:14:46,504] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:14:46,880] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:14:46,882] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:14:46,977] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:14:47,260] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:14:47,650] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:14:47,654] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:14:47,814] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:14:48,220] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:14:48,557] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:15:06,159] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:15:06,159] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 29.29369918699187, 'mem_free': 6705868800, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:15:06,159] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['kv'], 'cpu_utilization': 3.860711582134746, 'mem_free': 6859472896, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:15:06,159] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:15:06,162] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:15:06,290] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:15:06,676] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:15:07,024] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:15:07,037] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:15:07,167] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:15:07,557] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:15:07,904] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:15:07,911] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 52 100 20 100 32 7572 12116 --:--:-- --:--:-- --:--:-- 16000 [2021-05-11 18:15:07,931] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:15:08,058] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:15:08,521] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:15:08,949] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:15:09,092] - [cli_base:211] INFO - Test command with absolute path [2021-05-11 18:15:09,121] - [basetestcase:2379] INFO - **** add built-in 'default' user to node 172.23.105.151 **** [2021-05-11 18:15:09,168] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:15:09,169] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:15:09,292] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:15:14,298] - [basetestcase:2384] INFO - **** add 'admin' role to 'default' user **** [2021-05-11 18:15:14,378] - [importexporttest:921] INFO - test with absolute path [2021-05-11 18:15:14,411] - [importexporttest:953] INFO - load json to bucket default [2021-05-11 18:15:14,412] - [importexporttest:972] INFO - load sample data to bucket [2021-05-11 18:15:14,412] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbworkloadgen -n 172.23.105.151:8091 -u Administrator -p 'password' -j -i 1000 -b default [2021-05-11 18:15:14,931] - [remote_util:3473] INFO - command executed with root but got an error [' [####################] 100.0% (1053/estimated 1053 msgs)\x1b[A', '', 'bucket: default, msgs transferred...', ' : total | last | per sec', ' byte : 70107 | 70107 | 366100.5', 'done'] ... [2021-05-11 18:15:14,931] - [importexporttest:977] INFO - remove old export dir in /tmp/ [2021-05-11 18:15:14,932] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /tmp/export172.23.105.151 [2021-05-11 18:15:14,958] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:15:14,959] - [importexporttest:980] INFO - create export dir in /tmp/ [2021-05-11 18:15:14,959] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: mkdir /tmp/export172.23.105.151 [2021-05-11 18:15:15,020] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:15:15,332] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:15:15,435] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 18:15:15,443] - [rest_client:2363] INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute [2021-05-11 18:15:16,154] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: /opt/couchbase/bin/cbexport json -c http://172.23.105.151:8091 -u Administrator -p 'password' -b default -f list -o /tmp/export172.23.105.151/default -t 4 [2021-05-11 18:15:17,987] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:15:17,988] - [importexporttest:1550] INFO - Found 'successfully' in CLI output [2021-05-11 18:15:18,058] - [importexporttest:1461] INFO - copy export file from remote to local [2021-05-11 18:15:18,130] - [importexporttest:1468] INFO - compare 2 json files [2021-05-11 18:15:18,248] - [importexporttest:1522] INFO - export and sample json files are matched [2021-05-11 18:15:18,248] - [importexporttest:1527] INFO - remove file /tmp/export172.23.105.151 [2021-05-11 18:15:18,257] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:15:18,374] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:15:18,717] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:15:19,028] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:15:19,033] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: ok command: gen_server:cast(ns_cluster, leave). [2021-05-11 18:15:19,036] - [rest_client:1643] INFO - couchbase server is up but down soon. [2021-05-11 18:15:20,039] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:15:22,042] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:15:24,048] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:15:26,054] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:15:28,058] - [rest_client:1654] INFO - couchbase server is down. Waiting for couchbase server up [2021-05-11 18:15:30,068] - [rest_client:1651] INFO - couchbase server is up again in few seconds [2021-05-11 18:15:37,085] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:15:37,095] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:15:37,193] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:15:37,471] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:15:37,774] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:15:37,781] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:15:37,781] - [remote_util:5217] INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2021-05-11 18:15:38,799] - [rest_client:1081] INFO - quota for index service will be 256 MB [2021-05-11 18:15:38,799] - [rest_client:1083] INFO - set index quota to node 172.23.107.160 [2021-05-11 18:15:38,799] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:15:38,804] - [rest_client:1086] INFO - quota for fts service will be 512 MB [2021-05-11 18:15:38,804] - [rest_client:1088] INFO - set both index and fts quota at node 172.23.107.160 [2021-05-11 18:15:38,805] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:15:38,809] - [rest_client:1099] INFO - quota for kv: 4300 MB [2021-05-11 18:15:38,809] - [rest_client:1147] INFO - pools/default params : memoryQuota=4300 [2021-05-11 18:15:38,814] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,127.0.0.1,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 18:15:38,815] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.107.160: 8091:hostname=127.0.0.1%3A8091&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 18:15:38,843] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:15:38,843] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:15:38,933] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:15:39,233] - [task:769] INFO - adding node 172.23.107.184:8091 to cluster [2021-05-11 18:15:39,234] - [rest_client:1500] INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.107.160:8091 [2021-05-11 18:15:49,253] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:15:49,253] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:16:07,978] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:16:08,022] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:16:18,037] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:16:18,043] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.021243572235107 sec [2021-05-11 18:16:18,044] - [importexporttest:1067] INFO - add built-in user cbadminbucket to second cluster. [2021-05-11 18:16:18,066] - [rest_client:1022] ERROR - DELETE http://172.23.107.160:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:16:18,067] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:16:19,074] - [rest_client:2818] INFO - http://172.23.107.160:8091/pools/default/buckets with param: name=default&ramQuotaMB=250&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:16:19,109] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:16:19,109] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:16:19,999] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:20,139] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:20,387] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:20,612] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:20,768] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:16:20,768] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:16:21,116] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:21,324] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:21,656] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:21,833] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:22,071] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:16:22,071] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:16:22,401] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:22,551] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:22,843] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:22,985] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:23,114] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:16:23,115] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:16:23,395] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:23,548] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:23,846] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:23,987] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:24,180] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:16:24,181] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:16:24,401] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:24,544] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:24,850] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:24,995] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:25,214] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:16:25,214] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:16:25,440] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:25,645] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:25,871] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:16:26,039] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:16:26,220] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:16:26,220] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/cbimport json -c couchbase://172.23.107.160 -u Administrator -p 'password' -b default -d file:///tmp/export172.23.105.151/default -f list -g key::%index% [2021-05-11 18:16:26,459] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:16:26,461] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:16:26,589] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:16:26,987] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:16:27,338] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:16:27,350] - [cli_base:260] INFO - ======== clean up destination cluster ======= [2021-05-11 18:16:27,384] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.107.160 [2021-05-11 18:16:28,903] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:16:28,908] - [rest_client:137] INFO - node 172.23.107.160 existing buckets : [] [2021-05-11 18:16:28,908] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.107.160 [2021-05-11 18:16:28,917] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:16:28,927] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:16:28,942] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:16:28,944] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.107.184', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:16:28,990] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:16:28,993] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:16:49,012] - [rest_client:1833] INFO - rebalance progress took 20.02 seconds [2021-05-11 18:16:49,012] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:16:59,033] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.107.160 port:8091 ssh_username:root ? [('ns_1@172.23.107.184', 8091)] [2021-05-11 18:16:59,049] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:16:59,050] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.319132845979329, 'mem_free': 6719913984, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:16:59,050] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['kv'], 'cpu_utilization': 2.114803625377644, 'mem_free': 6862106624, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:16:59,050] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:16:59,053] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:16:59,186] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:16:59,731] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:00,158] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:00,161] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:17:00,310] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:17:00,860] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:01,240] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:01,243] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:17:01,387] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:17:01,676] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:01,989] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:01,992] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:17:02,128] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:17:02,556] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:02,911] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:20,852] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #13 test_export_and_import_back ============== [2021-05-11 18:17:20,888] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 18:17:22,182] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:17:22,186] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:17:22,186] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 18:17:22,194] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:17:22,202] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:17:22,209] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:17:22,216] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:22,228] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:17:22,231] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:17:22,273] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:17:22,276] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:17:42,306] - [rest_client:1833] INFO - rebalance progress took 20.03 seconds [2021-05-11 18:17:42,306] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:17:52,328] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 18:17:52,334] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:17:52,334] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,339] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:17:52,344] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:17:52,344] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,351] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:17:52,357] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:17:52,357] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,361] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:17:52,366] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:17:52,366] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,370] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:17:52,370] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #13 test_export_and_import_back ============== [2021-05-11 18:17:52,370] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 18:17:52,371] - [basetestcase:608] INFO - closing all memcached connections Cluster instance shutdown with force [2021-05-11 18:17:52,384] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,397] - [importexporttest:42] INFO - clean up server in import back tests [2021-05-11 18:17:52,405] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:17:52,412] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:17:52,416] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,431] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:17:52,431] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,435] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:17:52,440] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:17:52,440] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:52,443] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 summary so far suite security.x509tests.x509tests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest , pass 1 , fail 0 summary so far suite clitest.importexporttest.ImportExportTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_13 ok ---------------------------------------------------------------------- Ran 1 test in 246.947s OK test_ephemeral_buckets (buckettests.CreateBucketTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_14 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t buckettests.CreateBucketTests.test_ephemeral_buckets,default_bucket=false,sasl_buckets=2,bucket_size=100 Test Input params: {'default_bucket': 'false', 'sasl_buckets': '2', 'bucket_size': '100', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 14, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_14'} [2021-05-11 18:17:52,464] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:17:52,598] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:17:52,999] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:53,356] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:53,363] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:53,377] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:17:53,389] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 18:17:53,389] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #14 test_ephemeral_buckets============== [2021-05-11 18:17:53,410] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:17:53,419] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:17:53,427] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:17:53,437] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:17:53,442] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:53,461] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:17:53,462] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:53,466] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:17:53,471] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:17:53,471] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:53,475] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:17:53,480] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:17:53,480] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:53,484] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:17:53,488] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:17:53,488] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:17:53,492] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:17:53,492] - [basetestcase:295] INFO - initializing cluster [2021-05-11 18:17:54,469] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:17:54,473] - [task:157] INFO - {'uptime': '296', 'memoryTotal': 8192827392, 'memoryFree': 6752796672, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3419, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 18:17:54,479] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 18:17:54,479] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:17:54,479] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:17:54,487] - [task:201] INFO - quota for fts service will be 512 MB [2021-05-11 18:17:54,487] - [task:203] INFO - set both index and fts quota at node 172.23.105.151 [2021-05-11 18:17:54,487] - [rest_client:1159] INFO - pools/default params : ftsMemoryQuota=512 [2021-05-11 18:17:54,496] - [rest_client:1147] INFO - pools/default params : memoryQuota=3419 [2021-05-11 18:17:54,501] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'index', 'n1ql', 'fts', 'backup']) [2021-05-11 18:17:54,502] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup [2021-05-11 18:17:54,505] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091//node/controller/setupServices body: hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts%2Cbackup headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2021-05-11 18:17:54,505] - [rest_client:1130] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2021-05-11 18:17:54,505] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:17:54,505] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:17:54,591] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:17:54,593] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:17:54,732] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:17:55,135] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:55,488] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:55,502] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:17:55,566] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:17:55,567] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:17:55,571] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:55,573] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:55,578] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:17:55,598] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:17:55,603] - [task:157] INFO - {'uptime': '280', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 7811, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:17:55,605] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:17:55,606] - [rest_client:1147] INFO - pools/default params : memoryQuota=5233 [2021-05-11 18:17:55,611] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:17:55,611] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:17:55,706] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:17:55,708] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:17:55,843] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:17:56,297] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:56,665] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:56,676] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:17:56,739] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:17:56,740] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:17:56,742] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:56,744] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:56,750] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:17:56,771] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:17:56,775] - [task:157] INFO - {'uptime': '917', 'memoryTotal': 7934996480, 'memoryFree': 6660591616, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4300, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:17:56,780] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:17:56,787] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:17:56,787] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:17:56,887] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:17:56,890] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:17:57,008] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:17:57,289] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:57,596] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:57,608] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:17:57,809] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:17:57,810] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:17:57,814] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:57,816] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:57,821] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:17:57,848] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:17:57,855] - [task:157] INFO - {'uptime': '902', 'memoryTotal': 8192131072, 'memoryFree': 6977376256, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:17:57,857] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:17:57,857] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:17:57,865] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:17:57,865] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:17:57,961] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:17:57,963] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:17:58,093] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:17:58,528] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:17:58,882] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:17:58,897] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:17:58,955] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:17:58,955] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:17:58,962] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:58,967] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:17:58,977] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:17:59,003] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:17:59,128] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:18:04,131] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:18:04,147] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 18:18:04,149] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:18:04,289] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:18:04,701] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:05,047] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:05,057] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:18:05,122] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:18:05,122] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:18:05,122] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 18:18:06,020] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=bucket0&ramQuotaMB=100&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:18:06,050] - [rest_client:2843] INFO - 0.03 seconds to create bucket bucket0 [2021-05-11 18:18:06,078] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=bucket1&ramQuotaMB=100&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:18:06,118] - [rest_client:2843] INFO - 0.04 seconds to create bucket bucket1 [2021-05-11 18:18:06,119] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 18:18:07,167] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:07,404] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:07,585] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:18:07,586] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:18:07,948] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:08,200] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:08,325] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:18:08,325] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 18:18:08,597] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:08,803] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:08,981] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:18:08,981] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:18:09,170] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:09,431] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:09,562] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:18:09,562] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 18:18:09,832] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:10,049] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:10,230] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:18:10,231] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:18:10,454] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:10,722] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:10,830] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:18:10,830] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 18:18:11,065] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:11,257] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:11,418] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:18:11,419] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:18:11,598] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:11,830] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:11,936] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:18:11,936] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 18:18:12,170] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:12,359] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:12,527] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:18:12,527] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:18:12,716] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:12,950] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:13,056] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:18:13,056] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket0 in 172.23.105.151 to accept set ops [2021-05-11 18:18:13,295] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:13,483] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:13,595] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:18:13,595] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:18:13,831] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:14,067] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:14,175] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:18:14,176] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #14 test_ephemeral_buckets ============== [2021-05-11 18:18:14,185] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:18:14,312] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:18:14,684] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:15,029] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:15,032] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:18:15,170] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:18:15,660] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:16,028] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:16,031] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:18:16,128] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:18:16,418] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:16,723] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:16,727] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:18:16,856] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:18:17,260] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:17,595] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:35,796] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:18:35,796] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 29.05679513184584, 'mem_free': 6592557056, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:18:35,796] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:18:35,806] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:18:35,808] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:18:35,940] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:18:36,320] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:36,675] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:36,809] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=100&replicaNumber=1&bucketType=ephemeral&threadsNumber=3&flushEnabled=1&evictionPolicy=noEviction&compressionMode=passive [2021-05-11 18:18:36,842] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:18:36,842] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:18:37,710] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:38,049] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:38,227] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:18:38,227] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:18:38,448] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:38,728] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:38,856] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:18:38,857] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:18:39,119] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:39,348] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:39,468] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:18:39,468] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:18:39,752] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:40,031] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:40,158] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:18:40,158] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:18:40,375] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:40,675] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:40,790] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:18:40,791] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:18:41,050] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:41,320] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:41,443] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:18:41,445] - [bucket_helper:239] INFO - waiting for bucket creation to complete.... [2021-05-11 18:18:41,496] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['bucket0', 'bucket1', 'default'] [2021-05-11 18:18:41,497] - [rest_client:141] INFO - node 172.23.105.151 found bucket default [2021-05-11 18:18:41,605] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:41,723] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:41,988] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:42,543] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:42,548] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:42,590] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:42,599] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:43,008] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:43,015] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:43,045] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:43,065] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:43,391] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:43,412] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:18:43,440] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:18:43,504] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket0 [2021-05-11 18:18:44,857] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:18:44,881] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['bucket0', 'bucket1'] [2021-05-11 18:18:44,882] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:18:44,905] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['bucket0', 'bucket1'] [2021-05-11 18:18:44,921] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:18:44,921] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['backup', 'fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.584048460373549, 'mem_free': 6607085568, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:18:44,921] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:18:44,924] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:18:45,053] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:18:45,431] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:45,813] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:45,816] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:18:45,955] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:18:46,419] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:46,788] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:46,794] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:18:46,895] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:18:47,180] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:47,492] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:18:47,495] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:18:47,632] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:18:48,062] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:18:48,423] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:06,242] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #14 test_ephemeral_buckets ============== [2021-05-11 18:19:06,348] - [bucket_helper:133] INFO - deleting existing buckets ['bucket0', 'bucket1'] on 172.23.105.151 [2021-05-11 18:19:06,934] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:19:06,948] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['bucket1'] [2021-05-11 18:19:06,948] - [bucket_helper:156] INFO - deleted bucket : bucket0 from 172.23.105.151 [2021-05-11 18:19:07,555] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:19:07,560] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:19:07,560] - [bucket_helper:156] INFO - deleted bucket : bucket1 from 172.23.105.151 [2021-05-11 18:19:07,569] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:19:07,577] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:19:07,585] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:19:07,591] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:07,608] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:19:07,608] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:07,613] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:19:07,618] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:19:07,618] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:07,622] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:19:07,627] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:19:07,627] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:07,632] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:19:07,637] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:19:07,637] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:07,640] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:19:07,640] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #14 test_ephemeral_buckets ============== [2021-05-11 18:19:07,640] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 18:19:07,641] - [basetestcase:608] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 summary so far suite security.x509tests.x509tests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest , pass 1 , fail 0 summary so far suite clitest.importexporttest.ImportExportTests , pass 1 , fail 0 summary so far suite buckettests.CreateBucketTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_14 ok ---------------------------------------------------------------------- Ran 1 test in 75.187s OK test_end_to_end (tuqquery.n1ql_collections_end2end.QueryCollectionsEnd2EndTests) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_15 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t tuqquery.n1ql_collections_end2end.QueryCollectionsEnd2EndTests.test_end_to_end,GROUP=P0,test_name=e2e_1bucket_1scope_1collection,sanity_test=True,nodes_init=2,services_init=kv:n1ql:index-kv:n1ql:index,reset_services=True,bucket_size=100 Test Input params: {'GROUP': 'P0', 'test_name': 'e2e_1bucket_1scope_1collection', 'sanity_test': 'True', 'nodes_init': '2', 'services_init': 'kv:n1ql:index-kv:n1ql:index', 'reset_services': 'True', 'bucket_size': '100', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 15, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_15'} [2021-05-11 18:19:07,726] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:19:07,853] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:19:08,252] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:08,608] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:08,617] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:08,630] - [rest_client:2465] INFO - Node version in cluster 7.0.0-5148-enterprise [2021-05-11 18:19:08,640] - [rest_client:2475] INFO - Node versions in cluster ['7.0.0-5148-enterprise'] [2021-05-11 18:19:08,640] - [basetestcase:233] INFO - ============== basetestcase setup was started for test #15 test_end_to_end============== [2021-05-11 18:19:08,658] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:19:08,666] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:19:08,674] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:19:08,680] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:19:08,685] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:08,701] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:19:08,701] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:08,707] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:19:08,712] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:19:08,712] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:08,716] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:19:08,721] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:19:08,721] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:08,724] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:19:08,728] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:19:08,728] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:19:08,732] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:19:08,733] - [basetestcase:295] INFO - initializing cluster [2021-05-11 18:19:08,734] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:19:08,865] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:19:09,256] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:09,619] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:09,636] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:19:09,772] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:19:10,160] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:10,509] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:10,578] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:10,660] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:19:10,660] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service [2021-05-11 18:19:12,659] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:12,660] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:19:12,660] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:19:17,665] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:19:17,687] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:17,687] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:19:17,747] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:17,750] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:19:17,878] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:19:18,264] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:18,615] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:18,685] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:18,685] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:19:18,778] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:19:18,892] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:19:18,893] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service [2021-05-11 18:19:18,925] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:18,925] - [remote_util:338] INFO - 172.23.105.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:19:23,931] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:19:23,963] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:23,963] - [remote_util:4006] INFO - Couchbase server status: [' Process: 17853 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:19:23,963] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:19:24,043] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 18025 [2021-05-11 18:19:24,044] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:19:24,046] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:19:24,180] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:19:24,626] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:24,996] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:25,008] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:19:25,140] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:19:25,570] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:25,940] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:26,014] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:26,125] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:19:26,126] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service [2021-05-11 18:19:27,998] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:27,998] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:19:27,999] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:19:33,004] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:19:33,028] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:33,029] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:19:33,094] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:33,096] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:19:33,240] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:19:33,686] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:34,059] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:34,136] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:34,136] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:19:34,204] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:19:34,339] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:19:34,339] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service [2021-05-11 18:19:34,377] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:34,377] - [remote_util:338] INFO - 172.23.105.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:19:39,383] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:19:39,424] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:39,424] - [remote_util:4006] INFO - Couchbase server status: [' Process: 3435 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:19:39,424] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:19:39,519] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 3613 [2021-05-11 18:19:39,520] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:19:39,522] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:19:39,625] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:19:39,985] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:40,288] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:40,300] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:19:40,403] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:19:40,687] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:40,995] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:41,055] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:41,152] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:19:41,152] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl stop couchbase-server.service [2021-05-11 18:19:43,022] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:43,023] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:19:43,023] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:19:48,028] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:19:48,045] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:48,046] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:19:48,101] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:48,103] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:19:48,213] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:19:48,517] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:48,829] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:48,890] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:48,890] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:19:48,982] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:19:49,080] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:19:49,080] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl start couchbase-server.service [2021-05-11 18:19:49,119] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:49,120] - [remote_util:338] INFO - 172.23.107.160:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:19:54,125] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:19:54,159] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:54,159] - [remote_util:4006] INFO - Couchbase server status: [' Process: 22373 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:19:54,159] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:19:54,226] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 22515 [2021-05-11 18:19:54,227] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:19:54,228] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:19:54,371] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:19:54,786] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:55,145] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:55,167] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:19:55,300] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:19:55,685] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:19:56,023] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:19:56,052] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:19:56,129] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:19:56,130] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl stop couchbase-server.service [2021-05-11 18:19:58,014] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:19:58,015] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:19:58,015] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:20:03,020] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:20:03,043] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:03,043] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:20:03,101] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:03,102] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:20:03,236] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:20:03,647] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:20:03,985] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:20:04,056] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:20:04,056] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:20:04,107] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:20:04,225] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:20:04,226] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl start couchbase-server.service [2021-05-11 18:20:04,257] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:04,257] - [remote_util:338] INFO - 172.23.107.184:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:20:09,262] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:20:09,298] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:09,298] - [remote_util:4006] INFO - Couchbase server status: [' Process: 22692 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:20:09,298] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:20:09,371] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 22837 [2021-05-11 18:20:09,371] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:20:09,372] - [basetestcase:662] INFO - sleep for 10 secs. ... [2021-05-11 18:20:19,810] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:20:19,816] - [task:157] INFO - {'uptime': '57', 'memoryTotal': 8192827392, 'memoryFree': 6914805760, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4022, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 2000} [2021-05-11 18:20:19,821] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:20:19,821] - [task:196] INFO - quota for index service will be 256 MB [2021-05-11 18:20:19,821] - [task:198] INFO - set index quota to node 172.23.105.151 [2021-05-11 18:20:19,821] - [rest_client:1159] INFO - pools/default params : indexMemoryQuota=256 [2021-05-11 18:20:19,831] - [rest_client:1147] INFO - pools/default params : memoryQuota=3931 [2021-05-11 18:20:19,839] - [rest_client:1108] INFO - --> init_node_services(Administrator,password,172.23.105.151,8091,['kv', 'n1ql', 'index']) [2021-05-11 18:20:19,839] - [rest_client:1124] INFO - /node/controller/setupServices params on 172.23.105.151: 8091:hostname=172.23.105.151&user=Administrator&password=password&services=kv%2Cn1ql%2Cindex [2021-05-11 18:20:19,869] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:20:19,870] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:20:19,971] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:20:19,973] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:20:20,117] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:20:20,564] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:20:20,931] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:20:20,942] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:20:21,008] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:21,008] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:20:21,011] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:21,013] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:21,018] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:20:21,044] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:20:21,049] - [task:157] INFO - {'uptime': '42', 'memoryTotal': 8191283200, 'memoryFree': 6982905856, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:20:21,052] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:20:21,052] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 18:20:21,063] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:20:21,064] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:20:21,164] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:20:21,166] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:20:21,307] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:20:21,777] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:20:22,151] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:20:22,164] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:20:22,230] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:22,231] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:20:22,238] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:22,245] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:22,255] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:20:22,287] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:20:22,291] - [task:157] INFO - {'uptime': '26', 'memoryTotal': 7934996480, 'memoryFree': 6849724416, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:20:22,297] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:20:22,297] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:20:22,308] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:20:22,308] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:20:22,414] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:20:22,417] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:20:22,539] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:20:22,848] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:20:23,167] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:20:23,180] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:20:23,379] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:23,380] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:20:23,386] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:23,389] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:23,395] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:20:23,420] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:20:23,425] - [task:157] INFO - {'uptime': '16', 'memoryTotal': 8192131072, 'memoryFree': 6966607872, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:20:23,428] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:20:23,429] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:20:23,438] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:20:23,438] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:20:23,538] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:20:23,540] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:20:23,672] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:20:24,093] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:20:24,433] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:20:24,444] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:20:24,505] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:20:24,506] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:20:24,509] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:24,512] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:20:24,518] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:20:24,547] - [basetestcase:2379] INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.151 **** [2021-05-11 18:20:24,561] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:20:24,563] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:20:24,664] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:20:29,669] - [basetestcase:2384] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-05-11 18:20:29,685] - [basetestcase:332] INFO - done initializing cluster [2021-05-11 18:20:30,544] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 18:20:30,544] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 18:20:40,552] - [rest_client:1833] INFO - rebalance progress took 10.01 seconds [2021-05-11 18:20:40,552] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:20:59,382] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:20:59,432] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:21:09,446] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:21:09,453] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.020740985870361 sec [2021-05-11 18:21:09,456] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:21:09,591] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:21:10,002] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:10,352] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:10,362] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:21:10,423] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:21:10,423] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:21:10,424] - [basetestcase:2967] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 18:21:10,481] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=100&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:21:10,513] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:21:10,513] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:21:11,503] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:11,678] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:12,034] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:12,193] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:12,344] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:21:12,344] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:21:12,763] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:12,917] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:13,191] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:13,328] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:13,456] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:21:13,457] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:21:13,729] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:13,889] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:14,188] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:14,342] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:14,508] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:21:14,509] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:21:14,784] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:14,932] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:15,224] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:15,374] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:15,559] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:21:15,559] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:21:15,769] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:15,957] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:16,187] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:16,344] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:16,529] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:21:16,530] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:21:16,743] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:16,931] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:17,176] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:21:17,372] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:21:17,429] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:21:17,431] - [basetestcase:433] INFO - ============== basetestcase setup was finished for test #15 test_end_to_end ============== [2021-05-11 18:21:17,441] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:21:17,566] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:21:17,954] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:18,308] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:18,313] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:21:18,453] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:21:18,894] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:19,259] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:19,262] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:21:19,357] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:21:19,678] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:20,001] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:20,004] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:21:20,137] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:21:20,562] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:20,895] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:39,190] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:21:39,190] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 9.187279151943462, 'mem_free': 6726549504, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:21:39,190] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.11463664278403, 'mem_free': 6810087424, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:21:39,191] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:21:39,191] - [tuq:66] INFO - ============== QueryTests setup has started ============== [2021-05-11 18:21:39,193] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:21:39,335] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:21:39,764] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:40,145] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:40,170] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:21:40,300] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:21:40,709] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:21:41,072] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:21:41,073] - [tuq:1097] INFO - Generating default:default data... [2021-05-11 18:21:41,073] - [tuq:1103] INFO - -->start:generate_docs_employee for default... [2021-05-11 18:21:41,195] - [tuq:1105] INFO - -->end:generate_docs_employee for default... [2021-05-11 18:21:41,195] - [tuq:1200] INFO - Completed Generating default:default data... [2021-05-11 18:21:41,195] - [tuq:115] INFO - --->gens_load is done... [2021-05-11 18:21:41,196] - [tuq:123] INFO - -->starting rest connection... [2021-05-11 18:21:51,404] - [tuq:168] INFO - --> docs_per_day>0..generating TuqGenerators... [2021-05-11 18:22:02,000] - [tuq:170] INFO - --> End: docs_per_day>0..generating TuqGenerators... [2021-05-11 18:22:02,000] - [tuq:173] INFO - --> start: create_primary_index_for_3_0_and_greater... [2021-05-11 18:22:02,000] - [tuq:1754] INFO - --> start: ensure_primary_indexes_exist.. [2021-05-11 18:22:02,009] - [tuq:1441] INFO - RUN QUERY SELECT * FROM system:keyspaces WHERE keyspaces.`bucket` is missing [2021-05-11 18:22:02,009] - [rest_client:3950] INFO - query params : statement=SELECT+%2A+FROM+system%3Akeyspaces+WHERE+keyspaces.%60bucket%60+is+missing&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:02,044] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 33.456781ms [2021-05-11 18:22:02,045] - [tuq:1756] INFO - -->query_response:{'requestID': '10beb9c3-daf4-4f93-8673-7d9b6e5836ee', 'signature': {'*': '*'}, 'results': [{'keyspaces': {'datastore_id': 'http://127.0.0.1:8091', 'id': 'default', 'name': 'default', 'namespace': 'default', 'namespace_id': 'default', 'path': 'default:default'}}], 'status': 'success', 'metrics': {'elapsedTime': '33.456781ms', 'executionTime': '33.25402ms', 'resultCount': 1, 'resultSize': 158, 'serviceLoad': 1}} [2021-05-11 18:22:02,052] - [tuq:1441] INFO - RUN QUERY SELECT * FROM system:indexes WHERE indexes.bucket_id is missing [2021-05-11 18:22:02,052] - [rest_client:3950] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+WHERE+indexes.bucket_id+is+missing&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:02,258] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 202.825425ms [2021-05-11 18:22:02,259] - [tuq:1768] INFO - -->before create indexes: [{'name': '#primary', 'bucket': 'default', 'fields': [], 'state': 'online', 'using': 'gsi', 'is_primary': True}],[{'name': '#primary', 'bucket': 'default', 'fields': frozenset(), 'state': 'online', 'using': 'gsi', 'where': '', 'is_primary': True}],[] [2021-05-11 18:22:02,259] - [tuq:1043] INFO - -->Create desired indexes..desiredset=frozenset({frozenset({('bucket', 'default'), ('where', ''), ('name', '#primary'), ('fields', frozenset()), ('state', 'online'), ('using', 'gsi'), ('is_primary', True)})}),currentset=frozenset(),desiredindexes=[{'name': '#primary', 'bucket': 'default', 'fields': frozenset(), 'state': 'online', 'using': 'gsi', 'where': '', 'is_primary': True}] [2021-05-11 18:22:02,259] - [tuq:1048] INFO - creating index: default #primary gsi [2021-05-11 18:22:02,266] - [tuq:1441] INFO - RUN QUERY CREATE PRIMARY INDEX ON `default` USING gsi [2021-05-11 18:22:02,266] - [rest_client:3950] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+%60default%60+USING+gsi&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:04,618] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.349328781s [2021-05-11 18:22:04,626] - [tuq:1441] INFO - RUN QUERY SELECT * FROM system:indexes WHERE indexes.bucket_id is missing [2021-05-11 18:22:04,627] - [rest_client:3950] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+WHERE+indexes.bucket_id+is+missing&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:04,653] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 24.275086ms [2021-05-11 18:22:04,653] - [tuq:1770] INFO - --> end: ensure_primary_indexes_exist.. [2021-05-11 18:22:04,654] - [tuq:1748] INFO - -->waiting for indexes online, bucket:default [2021-05-11 18:22:04,660] - [tuq:1441] INFO - RUN QUERY SELECT * FROM system:indexes where name='#primary' [2021-05-11 18:22:04,660] - [rest_client:3950] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name%3D%27%23primary%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:04,665] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 1.813271ms [2021-05-11 18:22:04,666] - [tuq:175] INFO - --> End: create_primary_index_for_3_0_and_greater... [2021-05-11 18:22:04,666] - [tuq:195] INFO - ============== QueryTests setup has completed ============== [2021-05-11 18:22:04,666] - [n1ql_collections_end2end:213] INFO - ============== QueryCollectionsEnd2EndTests setup has started ============== [2021-05-11 18:22:04,672] - [tuq:1441] INFO - RUN QUERY SELECT * FROM system:indexes WHERE indexes.bucket_id is missing [2021-05-11 18:22:04,672] - [rest_client:3950] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+WHERE+indexes.bucket_id+is+missing&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:04,676] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 1.872195ms [2021-05-11 18:22:04,682] - [tuq:1441] INFO - RUN QUERY SELECT * FROM system:keyspaces WHERE keyspaces.`bucket` is missing [2021-05-11 18:22:04,682] - [rest_client:3950] INFO - query params : statement=SELECT+%2A+FROM+system%3Akeyspaces+WHERE+keyspaces.%60bucket%60+is+missing&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:04,687] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 1.73791ms [2021-05-11 18:22:04,687] - [tuq:309] INFO - ============== System Config: ============== [2021-05-11 18:22:04,693] - [tuq:1441] INFO - RUN QUERY SELECT COUNT(*) FROM `default` [2021-05-11 18:22:04,693] - [rest_client:3950] INFO - query params : statement=SELECT+COUNT%28%2A%29+FROM+%60default%60&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:04,710] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 15.475715ms [2021-05-11 18:22:04,710] - [tuq:317] INFO - Bucket: default [2021-05-11 18:22:04,711] - [tuq:318] INFO - Indexes: ['#primary'] [2021-05-11 18:22:04,711] - [tuq:319] INFO - Docs: 0 [2021-05-11 18:22:04,711] - [tuq:320] INFO - ============================================= [2021-05-11 18:22:04,711] - [n1ql_collections_end2end:220] INFO - ============== QueryCollectionsEnd2EndTests setup has completed ============== [2021-05-11 18:22:05,354] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=bucket1&ramQuotaMB=100&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:22:05,390] - [rest_client:2843] INFO - 0.04 seconds to create bucket bucket1 [2021-05-11 18:22:05,391] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:22:06,453] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:06,697] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:07,095] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:07,309] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:07,505] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:22:07,505] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:22:07,773] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:07,897] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:08,422] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:08,509] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:08,570] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:22:08,571] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:22:08,669] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:08,752] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:08,860] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:08,930] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:08,985] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:22:08,985] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:22:09,479] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:09,557] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:09,660] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:09,724] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:09,780] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:22:09,780] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:22:09,878] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:09,947] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:10,062] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:10,147] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:10,625] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:22:10,625] - [bucket_helper:335] INFO - waiting for memcached bucket : bucket1 in 172.23.105.151 to accept set ops [2021-05-11 18:22:10,720] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:10,783] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:10,885] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 bucket1 [2021-05-11 18:22:10,963] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 bucket1 [2021-05-11 18:22:11,030] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:22:11,032] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:22:11,163] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:22:11,571] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:22:11,929] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:22:12,057] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli collection-manage -c http://172.23.105.151:8091 -u Administrator -p password --bucket bucket1 --create-scope scope1 [2021-05-11 18:22:12,439] - [collections_cli_client:21] INFO - Scope created bucket1->scope1 [2021-05-11 18:22:12,441] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:22:12,599] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:22:13,039] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:22:13,406] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:22:13,538] - [remote_util:4695] INFO - command to run: /opt/couchbase/bin/couchbase-cli collection-manage -c http://172.23.105.151:8091 -u Administrator -p password --bucket bucket1 --create-collection scope1.collection1 [2021-05-11 18:22:13,928] - [collections_cli_client:13] INFO - Collection created bucket1->scope1->collection1 [2021-05-11 18:22:13,928] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:22:18,940] - [tuq:1441] INFO - RUN QUERY insert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id', {'val':1, 'name' : 'default:bucket1.scope1.collection1_name' }) [2021-05-11 18:22:18,940] - [rest_client:3950] INFO - query params : statement=insert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id%27%2C+%7B%27val%27%3A1%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:18,996] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 53.165599ms [2021-05-11 18:22:19,003] - [tuq:1441] INFO - RUN QUERY select * from default:bucket1.scope1.collection1 use keys ['default:bucket1.scope1.collection1_id'] [2021-05-11 18:22:19,004] - [rest_client:3950] INFO - query params : statement=select+%2A+from+default%3Abucket1.scope1.collection1+use+keys+%5B%27default%3Abucket1.scope1.collection1_id%27%5D&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:19,011] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 4.150273ms [2021-05-11 18:22:19,017] - [tuq:1441] INFO - RUN QUERY insert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id100', {'val':2, 'name' : 'default:bucket1.scope1.collection1_name' }) [2021-05-11 18:22:19,018] - [rest_client:3950] INFO - query params : statement=insert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id100%27%2C+%7B%27val%27%3A2%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:19,025] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.829103ms [2021-05-11 18:22:19,033] - [tuq:1441] INFO - RUN QUERY insert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id200', {'val':3, 'name' : 'default:bucket1.scope1.collection1_name' }) [2021-05-11 18:22:19,033] - [rest_client:3950] INFO - query params : statement=insert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id200%27%2C+%7B%27val%27%3A3%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:19,040] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 3.034571ms [2021-05-11 18:22:19,047] - [tuq:1441] INFO - RUN QUERY create primary index on default:bucket1.scope1.collection1 [2021-05-11 18:22:19,048] - [rest_client:3950] INFO - query params : statement=create+primary+index+on+default%3Abucket1.scope1.collection1&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:21,623] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.573464842s [2021-05-11 18:22:21,633] - [tuq:1441] INFO - RUN QUERY select count(*) from system:all_indexes where bucket_id='bucket1' and scope_id='scope1' and keyspace_id='collection1' [2021-05-11 18:22:21,633] - [rest_client:3950] INFO - query params : statement=select+count%28%2A%29+from+system%3Aall_indexes+where+bucket_id%3D%27bucket1%27+and+scope_id%3D%27scope1%27+and+keyspace_id%3D%27collection1%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:21,713] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 76.595503ms [2021-05-11 18:22:21,723] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=2 [2021-05-11 18:22:21,723] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D2&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:21,734] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 6.504204ms [2021-05-11 18:22:21,743] - [tuq:1441] INFO - RUN QUERY update default:bucket1.scope1.collection1 set name=name||'_updated' where val=2 [2021-05-11 18:22:21,743] - [rest_client:3950] INFO - query params : statement=update+default%3Abucket1.scope1.collection1+set+name%3Dname%7C%7C%27_updated%27+where+val%3D2&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:21,758] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 8.954749ms [2021-05-11 18:22:21,765] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=2 [2021-05-11 18:22:21,766] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D2&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:22,820] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 1.049710355s [2021-05-11 18:22:22,830] - [tuq:1441] INFO - RUN QUERY upsert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id', {'val':2, 'name' : 'default:bucket1.scope1.collection1_name' }) [2021-05-11 18:22:22,830] - [rest_client:3950] INFO - query params : statement=upsert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id%27%2C+%7B%27val%27%3A2%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:22,835] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.464276ms [2021-05-11 18:22:22,843] - [tuq:1441] INFO - RUN QUERY select * from default:bucket1.scope1.collection1 where meta().id='default:bucket1.scope1.collection1_id' [2021-05-11 18:22:22,844] - [rest_client:3950] INFO - query params : statement=select+%2A+from+default%3Abucket1.scope1.collection1+where+meta%28%29.id%3D%27default%3Abucket1.scope1.collection1_id%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:23,139] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 292.932884ms [2021-05-11 18:22:23,145] - [tuq:1441] INFO - RUN QUERY upsert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id2', {'val':2, 'name' : 'default:bucket1.scope1.collection1_name2' }) [2021-05-11 18:22:23,145] - [rest_client:3950] INFO - query params : statement=upsert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id2%27%2C+%7B%27val%27%3A2%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name2%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:23,149] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.283106ms [2021-05-11 18:22:23,156] - [tuq:1441] INFO - RUN QUERY select * from default:bucket1.scope1.collection1 where meta().id='default:bucket1.scope1.collection1_id2' [2021-05-11 18:22:23,156] - [rest_client:3950] INFO - query params : statement=select+%2A+from+default%3Abucket1.scope1.collection1+where+meta%28%29.id%3D%27default%3Abucket1.scope1.collection1_id2%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:23,166] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 5.680628ms [2021-05-11 18:22:23,172] - [tuq:1441] INFO - RUN QUERY drop primary index on default:bucket1.scope1.collection1 [2021-05-11 18:22:23,172] - [rest_client:3950] INFO - query params : statement=drop+primary+index+on+default%3Abucket1.scope1.collection1&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:23,233] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 57.913616ms [2021-05-11 18:22:23,238] - [tuq:1441] INFO - RUN QUERY select count(*) from system:all_indexes where bucket_id='bucket1' and scope_id='scope1' and keyspace_id='collection1' [2021-05-11 18:22:23,238] - [rest_client:3950] INFO - query params : statement=select+count%28%2A%29+from+system%3Aall_indexes+where+bucket_id%3D%27bucket1%27+and+scope_id%3D%27scope1%27+and+keyspace_id%3D%27collection1%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:23,267] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 24.188952ms [2021-05-11 18:22:23,273] - [tuq:1441] INFO - RUN QUERY create index idx_val on default:bucket1.scope1.collection1(val) [2021-05-11 18:22:23,273] - [rest_client:3950] INFO - query params : statement=create+index+idx_val+on+default%3Abucket1.scope1.collection1%28val%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:25,432] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.157068733s [2021-05-11 18:22:25,454] - [tuq:1441] INFO - RUN QUERY select count(*) from system:all_indexes where bucket_id='bucket1' and scope_id='scope1' and keyspace_id='collection1' and name='idx_val' [2021-05-11 18:22:25,454] - [rest_client:3950] INFO - query params : statement=select+count%28%2A%29+from+system%3Aall_indexes+where+bucket_id%3D%27bucket1%27+and+scope_id%3D%27scope1%27+and+keyspace_id%3D%27collection1%27+and+name%3D%27idx_val%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:25,489] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 30.716696ms [2021-05-11 18:22:25,495] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=3 [2021-05-11 18:22:25,495] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:25,503] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 5.0732ms [2021-05-11 18:22:25,508] - [tuq:1441] INFO - RUN QUERY update default:bucket1.scope1.collection1 set name=name||'_updated' where val=3 [2021-05-11 18:22:25,508] - [rest_client:3950] INFO - query params : statement=update+default%3Abucket1.scope1.collection1+set+name%3Dname%7C%7C%27_updated%27+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:25,526] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 12.798755ms [2021-05-11 18:22:25,536] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=3 [2021-05-11 18:22:25,536] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:25,717] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 175.941874ms [2021-05-11 18:22:25,723] - [tuq:1441] INFO - RUN QUERY upsert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id', {'val':3, 'name' : 'default:bucket1.scope1.collection1_name_updated_x3' }) [2021-05-11 18:22:25,723] - [rest_client:3950] INFO - query params : statement=upsert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id%27%2C+%7B%27val%27%3A3%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name_updated_x3%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:25,729] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.583701ms [2021-05-11 18:22:25,735] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=3 [2021-05-11 18:22:25,735] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:26,970] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 1.230934171s [2021-05-11 18:22:26,975] - [tuq:1441] INFO - RUN QUERY upsert into default:bucket1.scope1.collection1 (KEY, VALUE) VALUES ('default:bucket1.scope1.collection1_id3', {'val':33, 'name' : 'default:bucket1.scope1.collection1_name3' }) [2021-05-11 18:22:26,976] - [rest_client:3950] INFO - query params : statement=upsert+into+default%3Abucket1.scope1.collection1+%28KEY%2C+VALUE%29+VALUES+%28%27default%3Abucket1.scope1.collection1_id3%27%2C+%7B%27val%27%3A33%2C+%27name%27+%3A+%27default%3Abucket1.scope1.collection1_name3%27+%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:26,981] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 2.643235ms [2021-05-11 18:22:26,987] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=33 [2021-05-11 18:22:26,987] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D33&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:26,999] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 4.248206ms [2021-05-11 18:22:27,004] - [tuq:1441] INFO - RUN QUERY MERGE INTO default:bucket1.scope1.collection1 AS target USING [ {'name':'default:bucket1.scope1.collection1_name4', 'val': 4} ] AS source ON target.val = source.val WHEN MATCHED THEN UPDATE SET target.name = source.name WHEN NOT MATCHED THEN INSERT (KEY UUID(), VALUE {'name': source.name, 'val': source.val}) [2021-05-11 18:22:27,005] - [rest_client:3950] INFO - query params : statement=MERGE+INTO+default%3Abucket1.scope1.collection1+AS+target+USING+%5B+%7B%27name%27%3A%27default%3Abucket1.scope1.collection1_name4%27%2C+%27val%27%3A+4%7D+%5D+AS+source+ON+target.val+%3D+source.val+WHEN+MATCHED+THEN+UPDATE+SET+target.name+%3D+source.name+WHEN+NOT+MATCHED+THEN+INSERT+%28KEY+UUID%28%29%2C+VALUE+%7B%27name%27%3A+source.name%2C+%27val%27%3A+source.val%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,023] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 9.426302ms [2021-05-11 18:22:27,028] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=4 [2021-05-11 18:22:27,028] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D4&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,149] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 114.667189ms [2021-05-11 18:22:27,156] - [tuq:1441] INFO - RUN QUERY MERGE INTO default:bucket1.scope1.collection1 AS target USING [ {'name':'default:bucket1.scope1.collection1_merged_name', 'val': 3} ] AS source ON target.val = source.val WHEN MATCHED THEN UPDATE SET target.name = source.name WHEN NOT MATCHED THEN INSERT (KEY UUID(), VALUE {'name': source.name, 'val': source.val}) [2021-05-11 18:22:27,156] - [rest_client:3950] INFO - query params : statement=MERGE+INTO+default%3Abucket1.scope1.collection1+AS+target+USING+%5B+%7B%27name%27%3A%27default%3Abucket1.scope1.collection1_merged_name%27%2C+%27val%27%3A+3%7D+%5D+AS+source+ON+target.val+%3D+source.val+WHEN+MATCHED+THEN+UPDATE+SET+target.name+%3D+source.name+WHEN+NOT+MATCHED+THEN+INSERT+%28KEY+UUID%28%29%2C+VALUE+%7B%27name%27%3A+source.name%2C+%27val%27%3A+source.val%7D%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,172] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 12.445887ms [2021-05-11 18:22:27,178] - [tuq:1441] INFO - RUN QUERY select name from default:bucket1.scope1.collection1 where val=3 [2021-05-11 18:22:27,178] - [rest_client:3950] INFO - query params : statement=select+name+from+default%3Abucket1.scope1.collection1+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,187] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 7.075482ms [2021-05-11 18:22:27,193] - [tuq:1441] INFO - RUN QUERY delete from default:bucket1.scope1.collection1 where val=3 [2021-05-11 18:22:27,193] - [rest_client:3950] INFO - query params : statement=delete+from+default%3Abucket1.scope1.collection1+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,206] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 10.846288ms [2021-05-11 18:22:27,212] - [tuq:1441] INFO - RUN QUERY select count(*) from default:bucket1.scope1.collection1 where val=3 [2021-05-11 18:22:27,212] - [rest_client:3950] INFO - query params : statement=select+count%28%2A%29+from+default%3Abucket1.scope1.collection1+where+val%3D3&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,220] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 5.793575ms [2021-05-11 18:22:27,235] - [tuq:1441] INFO - RUN QUERY ALTER INDEX default:bucket1.scope1.collection1.idx_val WITH {'action': 'replica_count', 'num_replica': 1} [2021-05-11 18:22:27,236] - [rest_client:3950] INFO - query params : statement=ALTER+INDEX+default%3Abucket1.scope1.collection1.idx_val+WITH+%7B%27action%27%3A+%27replica_count%27%2C+%27num_replica%27%3A+1%7D&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,367] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 128.359322ms [2021-05-11 18:22:27,379] - [tuq:1441] INFO - RUN QUERY BUILD INDEX ON default:bucket1.scope1.collection1(idx_val) [2021-05-11 18:22:27,379] - [rest_client:3950] INFO - query params : statement=BUILD+INDEX+ON+default%3Abucket1.scope1.collection1%28idx_val%29&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,401] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 16.481931ms [2021-05-11 18:22:27,407] - [tuq:1441] INFO - RUN QUERY drop index idx_val on default:bucket1.scope1.collection1 [2021-05-11 18:22:27,407] - [rest_client:3950] INFO - query params : statement=drop+index+idx_val+on+default%3Abucket1.scope1.collection1&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,460] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 48.63467ms [2021-05-11 18:22:27,467] - [tuq:1441] INFO - RUN QUERY select count(*) from system:all_indexes where bucket_id='bucket1' and scope_id='scope1' and keyspace_id='collection1' and name='idx_val' [2021-05-11 18:22:27,467] - [rest_client:3950] INFO - query params : statement=select+count%28%2A%29+from+system%3Aall_indexes+where+bucket_id%3D%27bucket1%27+and+scope_id%3D%27scope1%27+and+keyspace_id%3D%27collection1%27+and+name%3D%27idx_val%27&scan_consistency=REQUEST_PLUS [2021-05-11 18:22:27,506] - [tuq:1493] INFO - TOTAL ELAPSED TIME: 35.431878ms [2021-05-11 18:22:27,507] - [n1ql_collections_end2end:228] INFO - ============== QueryCollectionsEnd2EndTests tearDown has started ============== [2021-05-11 18:22:27,537] - [bucket_helper:133] INFO - deleting existing buckets ['bucket1', 'default'] on 172.23.105.151 [2021-05-11 18:22:29,234] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:22:29,249] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : ['default'] [2021-05-11 18:22:29,250] - [bucket_helper:156] INFO - deleted bucket : bucket1 from 172.23.105.151 [2021-05-11 18:22:30,548] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:22:30,561] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:22:30,561] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 18:22:30,575] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:22:30,583] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:22:30,592] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:22:30,593] - [tuq:265] INFO - ============== QueryTests tearDown has started ============== [2021-05-11 18:22:30,593] - [tuq:281] INFO - ============== QueryTests tearDown has completed ============== [2021-05-11 18:22:30,625] - [basetestcase:466] INFO - ------- Cluster statistics ------- [2021-05-11 18:22:30,626] - [basetestcase:468] INFO - 172.23.105.151:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.98739387702598, 'mem_free': 6443790336, 'mem_total': 8192827392, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:22:30,626] - [basetestcase:468] INFO - 172.23.105.153:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 37.66684463427656, 'mem_free': 6546763776, 'mem_total': 8191283200, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-05-11 18:22:30,626] - [basetestcase:469] INFO - --- End of cluster statistics --- [2021-05-11 18:22:30,629] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:22:30,772] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:22:31,183] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:22:31,534] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:22:31,536] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:22:31,669] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:22:32,115] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:22:32,490] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:22:32,493] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:22:32,596] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:22:32,895] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:22:33,221] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:22:33,224] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:22:33,358] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:22:33,767] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:22:34,112] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:22:53,288] - [basetestcase:571] INFO - ============== basetestcase cleanup was started for test #15 test_end_to_end ============== [2021-05-11 18:22:53,317] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:22:53,329] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:22:53,341] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:22:53,349] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:22:53,355] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:22:53,371] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:22:53,373] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:22:53,405] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:22:53,408] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:23:03,423] - [rest_client:1894] INFO - rebalance percentage : 66.00 % [2021-05-11 18:23:13,438] - [rest_client:1894] INFO - rebalance percentage : 66.00 % [2021-05-11 18:23:33,467] - [rest_client:1833] INFO - rebalance progress took 40.06 seconds [2021-05-11 18:23:33,467] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:23:43,493] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 18:23:43,505] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:23:43,505] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:23:43,512] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:23:43,523] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:23:43,523] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:23:43,535] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:23:43,546] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:23:43,546] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:23:43,553] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:23:43,559] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:23:43,559] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:23:43,563] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:23:43,563] - [basetestcase:595] INFO - ============== basetestcase cleanup was finished for test #15 test_end_to_end ============== [2021-05-11 18:23:43,563] - [basetestcase:603] INFO - closing all ssh connections [2021-05-11 18:23:43,564] - [basetestcase:608] INFO - closing all memcached connections [2021-05-11 18:23:43,566] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:23:43,748] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:23:44,188] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:23:44,569] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:23:44,589] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:23:44,718] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:23:45,160] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:23:45,613] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:23:45,717] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:23:45,839] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:23:45,839] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl stop couchbase-server.service [2021-05-11 18:23:47,612] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:23:47,612] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:23:47,613] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:23:52,618] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:23:52,663] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:23:52,663] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:23:52,750] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:23:52,752] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:23:52,900] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:23:53,317] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:23:53,750] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:23:53,827] - [remote_util:964] INFO - 172.23.105.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:23:53,828] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:23:53,903] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:23:54,052] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:23:54,052] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl start couchbase-server.service [2021-05-11 18:23:54,084] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:23:54,084] - [remote_util:338] INFO - 172.23.105.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:23:59,090] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:23:59,127] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:23:59,127] - [remote_util:4006] INFO - Couchbase server status: [' Process: 23901 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:23:59,127] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:23:59,218] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.151: with pid 24073 [2021-05-11 18:23:59,218] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:23:59,221] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:23:59,366] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:23:59,858] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:00,266] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:00,287] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:24:00,435] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:24:00,859] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:01,278] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:01,325] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:24:01,502] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:24:01,503] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl stop couchbase-server.service [2021-05-11 18:24:03,411] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:03,412] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:24:03,412] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:24:08,418] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:24:08,456] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:08,456] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:24:08,528] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:08,530] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:24:08,679] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:24:09,144] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:09,520] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:09,558] - [remote_util:964] INFO - 172.23.105.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:24:09,558] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:24:09,635] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:24:09,778] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:24:09,779] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl start couchbase-server.service [2021-05-11 18:24:09,813] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:09,814] - [remote_util:338] INFO - 172.23.105.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:24:14,819] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:24:14,861] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:14,862] - [remote_util:4006] INFO - Couchbase server status: [' Process: 8799 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:24:14,862] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:24:14,950] - [remote_util:176] INFO - process beam.smp is running on 172.23.105.153: with pid 9009 [2021-05-11 18:24:14,951] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:24:14,954] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:24:15,050] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:24:15,335] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:15,669] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:15,683] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:24:15,986] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:24:16,294] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:16,620] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:16,690] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:24:16,779] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:24:16,779] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl stop couchbase-server.service [2021-05-11 18:24:18,595] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:18,595] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:24:18,596] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:24:23,601] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:24:23,620] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:23,621] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:24:23,679] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:23,681] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:24:23,780] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:24:24,076] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:24,410] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:24,434] - [remote_util:964] INFO - 172.23.107.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:24:24,435] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:24:24,529] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:24:24,628] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:24:24,628] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl start couchbase-server.service [2021-05-11 18:24:24,667] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:24,667] - [remote_util:338] INFO - 172.23.107.160:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:24:29,673] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:24:29,709] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:29,710] - [remote_util:4006] INFO - Couchbase server status: [' Process: 26981 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:24:29,710] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:24:29,796] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.160: with pid 27123 [2021-05-11 18:24:29,796] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:24:29,798] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:24:29,940] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:24:30,404] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:30,749] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:30,763] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:24:30,905] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:24:31,316] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:31,664] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:31,737] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:24:31,843] - [remote_util:3960] INFO - Running systemd command on this server [2021-05-11 18:24:31,843] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl stop couchbase-server.service [2021-05-11 18:24:33,573] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:33,573] - [basetestcase:2416] INFO - Couchbase stopped [2021-05-11 18:24:33,574] - [basetestcase:662] INFO - sleep for 5 secs. ... [2021-05-11 18:24:38,579] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-05-11 18:24:38,602] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:38,602] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-05-11 18:24:38,664] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:38,666] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:24:38,799] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:24:39,195] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:39,541] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:39,612] - [remote_util:964] INFO - 172.23.107.184 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-05-11 18:24:39,612] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:24:39,711] - [remote_util:3979] INFO - Starting couchbase server [2021-05-11 18:24:39,866] - [remote_util:4001] INFO - Running systemd command on this server [2021-05-11 18:24:39,866] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl start couchbase-server.service [2021-05-11 18:24:39,895] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:39,895] - [remote_util:338] INFO - 172.23.107.184:sleep for 5 secs. waiting for couchbase server to come up ... [2021-05-11 18:24:44,900] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-05-11 18:24:44,935] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:24:44,935] - [remote_util:4006] INFO - Couchbase server status: [' Process: 27502 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-05-11 18:24:44,935] - [remote_util:157] INFO - Checking for process beam.smp on linux [2021-05-11 18:24:45,014] - [remote_util:176] INFO - process beam.smp is running on 172.23.107.184: with pid 27646 [2021-05-11 18:24:45,014] - [basetestcase:2430] INFO - Couchbase started [2021-05-11 18:24:45,015] - [basetestcase:662] INFO - sleep for 10 secs. ... Cluster instance shutdown with force [2021-05-11 18:24:55,037] - [n1ql_collections_end2end:231] INFO - ============== QueryCollectionsEnd2EndTests tearDown has completed ============== summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 summary so far suite security.x509tests.x509tests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest , pass 1 , fail 0 summary so far suite clitest.importexporttest.ImportExportTests , pass 1 , fail 0 summary so far suite buckettests.CreateBucketTests , pass 1 , fail 0 summary so far suite tuqquery.n1ql_collections_end2end.QueryCollectionsEnd2EndTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_15 ok ---------------------------------------------------------------------- Ran 1 test in 347.315s OK load_with_ops (xdcr.uniXDCR.unidirectional) ... Logs will be stored at /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_16 ./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t xdcr.uniXDCR.unidirectional.load_with_ops,items=5000,ctopology=chain,rdirection=unidirection,update=C1,delete=C1,java_sdk_client=True,collection_density=medium Test Input params: {'items': '5000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'update': 'C1', 'delete': 'C1', 'java_sdk_client': 'True', 'collection_density': 'medium', 'ini': 'node_conf.ini', 'cluster_name': 'node_conf', 'spec': 'py-multi-node-sanity-1', 'conf_file': 'conf/py-multi-node-sanity-1.conf', 'get-cbcollect-info': 'True', 'get-couch-dbinfo': 'True', 'skip_cleanup': 'False', 'skip_log_scan': 'False', 'skip_security_scan': 'False', 'num_nodes': 4, 'case_number': 16, 'logs_folder': '/home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_16'} [2021-05-11 18:24:55,458] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password [2021-05-11 18:24:55,458] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 18:24:55,460] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:24:55,588] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:24:55,974] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:56,328] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:56,329] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:24:56,430] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:24:56,431] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:24:56,433] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:24:56,559] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:24:56,931] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:57,277] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:57,277] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:24:57,376] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:24:57,377] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:24:57,378] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:24:57,524] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:24:57,978] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:58,362] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:58,363] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:24:58,434] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:24:58,435] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:24:58,437] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:24:58,588] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:24:59,055] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:24:59,452] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:24:59,453] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:24:59,564] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:24:59,564] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:24:59,567] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:24:59,670] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:25:00,021] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:00,339] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:00,340] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:00,424] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:00,425] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:00,426] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:25:00,540] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:25:00,830] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:01,165] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:01,166] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:01,256] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:01,256] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:01,258] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:25:01,397] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:25:01,816] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:02,167] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:02,168] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "panic:" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:02,268] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:02,268] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:02,270] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:25:02,398] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:25:02,820] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:03,150] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:03,151] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:03,241] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:03,242] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:03,242] - [xdcrnewbasetests:2981] INFO - {'172.23.105.151': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.105.153': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.160': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.184': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}} [2021-05-11 18:25:03,250] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/diag/eval/ body: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password [2021-05-11 18:25:03,250] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: False content: API is accessible from localhost only command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 18:25:03,253] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:25:03,382] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:25:03,765] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:04,112] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:04,113] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:04,211] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:04,211] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:04,215] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:25:04,358] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:25:04,817] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:05,201] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:05,202] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:05,266] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:05,267] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:05,270] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:25:05,380] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:25:05,698] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:06,038] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:06,039] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:06,135] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:06,136] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:06,138] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:25:06,261] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:25:06,662] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:06,990] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:06,991] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "Try to fix Pipeline" API is accessible from localhost only/goxdcr.log* [2021-05-11 18:25:07,087] - [remote_util:3473] INFO - command executed with root but got an error ['gzip: API.gz: No such file or directory', 'gzip: is.gz: No such file or directory', 'gzip: accessible.gz: No such file or directory', 'gzip: from.gz: No such file or directory', 'gzip: localhost.gz: No such file or directory', 'gzip: only/goxdcr.log*.gz: No such file or directory'] ... [2021-05-11 18:25:07,088] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:25:07,088] - [xdcrnewbasetests:2999] INFO - {'172.23.105.151': 0, '172.23.105.153': 0, '172.23.107.160': 0, '172.23.107.184': 0} [2021-05-11 18:25:11,891] - [xdcrnewbasetests:2741] INFO - ==== XDCRNewbasetests setup is started for test #16 load_with_ops ==== [2021-05-11 18:25:11,901] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:11,914] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:11,914] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 18:25:11,942] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root] [2021-05-11 18:25:11,954] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:25:11,958] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:11,976] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:25:11,976] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:11,979] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:25:11,998] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:25:12,002] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:12,021] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:25:12,021] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:12,025] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:25:12,025] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 18:25:12,050] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 18:25:12,061] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:25:12,065] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:12,083] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:25:12,083] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:12,087] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:25:12,103] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:25:12,107] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:12,125] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:25:12,125] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:25:12,129] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:25:12,129] - [xdcrnewbasetests:3009] INFO - Initializing all clusters... [2021-05-11 18:25:12,137] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:25:12,265] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:25:12,667] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:13,037] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:13,049] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:13,049] - [remote_util:5217] INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2021-05-11 18:25:13,050] - [xdcrnewbasetests:1241] INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2021-05-11 18:25:13,055] - [rest_client:1022] ERROR - POST http://172.23.105.151:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password [2021-05-11 18:25:13,055] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:13,901] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:25:13,908] - [task:157] INFO - {'uptime': '77', 'memoryTotal': 8192827392, 'memoryFree': 6906650624, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4022, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 18:25:13,910] - [rest_client:1022] ERROR - GET http://172.23.105.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:13,911] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:25:13,925] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:25:13,925] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:25:14,017] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:25:14,018] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:25:14,167] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:25:14,566] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:14,928] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:14,944] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:25:15,000] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:25:15,001] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:25:15,003] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:15,006] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:15,010] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:25:15,034] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:25:15,038] - [task:157] INFO - {'uptime': '62', 'memoryTotal': 8191283200, 'memoryFree': 6985650176, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:25:15,041] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:15,041] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 18:25:15,049] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:25:15,049] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:25:15,141] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:25:15,143] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:25:15,275] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:25:15,730] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:16,110] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:16,123] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:25:16,190] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:25:16,190] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:25:16,193] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:16,196] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:16,201] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:25:17,221] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 18:25:17,221] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 18:25:27,245] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:25:27,245] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:25:46,399] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:25:46,431] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:25:56,445] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:25:56,454] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.022428035736084 sec [2021-05-11 18:25:56,462] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:25:56,558] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:25:56,827] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:57,131] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:57,138] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:57,139] - [remote_util:5217] INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2021-05-11 18:25:57,139] - [xdcrnewbasetests:1241] INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2021-05-11 18:25:57,141] - [rest_client:1022] ERROR - POST http://172.23.107.160:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'API is accessible from localhost only' b'API is accessible from localhost only' auth: Administrator:password [2021-05-11 18:25:57,141] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:57,954] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:25:57,958] - [task:157] INFO - {'uptime': '87', 'memoryTotal': 7934996480, 'memoryFree': 6846271488, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 3874, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:25:57,960] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:57,960] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:25:57,968] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:25:57,968] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:25:58,126] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:25:58,129] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:25:58,228] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:25:58,519] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:25:58,822] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:25:58,838] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:25:59,035] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:25:59,035] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:25:59,038] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:59,040] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:25:59,046] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:25:59,069] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:25:59,073] - [task:157] INFO - {'uptime': '72', 'memoryTotal': 8192131072, 'memoryFree': 6976798720, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:25:59,076] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:25:59,077] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:25:59,085] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:25:59,085] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:25:59,178] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:25:59,179] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:25:59,313] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:25:59,735] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:26:00,101] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:26:00,113] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:26:00,173] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:26:00,174] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:26:00,178] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:26:00,182] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:26:00,190] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:26:01,210] - [task:769] INFO - adding node 172.23.107.184:8091 to cluster [2021-05-11 18:26:01,211] - [rest_client:1500] INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.107.160:8091 [2021-05-11 18:26:11,228] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:26:11,228] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:26:29,988] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:26:30,022] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:26:40,036] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:26:40,043] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.021038055419922 sec [2021-05-11 18:26:40,043] - [xdcrnewbasetests:2827] INFO - Adding user: cbadminbucket with roles: admin [2021-05-11 18:26:40,067] - [rest_client:1022] ERROR - DELETE http://172.23.105.151:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:26:40,067] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:26:40,190] - [xdcrnewbasetests:2827] INFO - Adding user: cbadminbucket with roles: admin [2021-05-11 18:26:40,211] - [rest_client:1022] ERROR - DELETE http://172.23.107.160:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-05-11 18:26:40,212] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-05-11 18:26:40,526] - [rest_client:2818] INFO - http://172.23.105.151:8091/pools/default/buckets with param: name=default&ramQuotaMB=3686&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:26:40,559] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:26:40,560] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:26:41,316] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:41,451] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:41,763] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:41,922] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:42,141] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:26:42,141] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:26:42,424] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:42,580] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:42,906] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:43,084] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:43,317] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:26:43,317] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:26:43,553] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:43,776] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:44,003] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:44,208] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:44,341] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:26:44,342] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:26:44,551] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:44,762] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:45,017] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:45,228] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:45,400] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:26:45,400] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:26:45,712] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:45,892] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:46,147] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:46,363] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:46,514] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:26:46,514] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.105.151 to accept set ops [2021-05-11 18:26:46,808] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:46,957] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:47,237] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:26:47,385] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:26:47,529] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:26:48,079] - [rest_client:2818] INFO - http://172.23.107.160:8091/pools/default/buckets with param: name=default&ramQuotaMB=3555&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-05-11 18:26:48,110] - [rest_client:2843] INFO - 0.03 seconds to create bucket default [2021-05-11 18:26:48,110] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:26:48,936] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:49,083] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:49,382] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:49,554] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:49,780] - [task:380] WARNING - vbucket map not ready after try 0 [2021-05-11 18:26:49,780] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:26:50,081] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:50,270] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:50,634] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:50,809] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:51,040] - [task:380] WARNING - vbucket map not ready after try 1 [2021-05-11 18:26:51,040] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:26:51,280] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:51,479] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:51,693] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:51,887] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:52,015] - [task:380] WARNING - vbucket map not ready after try 2 [2021-05-11 18:26:52,016] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:26:52,232] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:52,436] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:52,658] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:52,871] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:53,009] - [task:380] WARNING - vbucket map not ready after try 3 [2021-05-11 18:26:53,010] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:26:53,288] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:53,424] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:53,659] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:53,871] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:54,007] - [task:380] WARNING - vbucket map not ready after try 4 [2021-05-11 18:26:54,007] - [bucket_helper:335] INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2021-05-11 18:26:54,294] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:54,447] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:54,727] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:26:54,879] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:26:55,033] - [task:380] WARNING - vbucket map not ready after try 5 [2021-05-11 18:26:55,035] - [xdcrnewbasetests:2747] INFO - ==== XDCRNewbasetests setup is finished for test #16 load_with_ops ==== [2021-05-11 18:26:55,041] - [rest_client:1361] INFO - adding remote cluster hostname:172.23.107.160:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.105.151:8091 [2021-05-11 18:27:05,083] - [rest_client:1417] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {} [2021-05-11 18:27:05,121] - [rest_client:1432] INFO - Replication created with id: 6278ccceb92e6cf9291193703f7681ec/default/default [2021-05-11 18:27:15,151] - [rest_client:3156] INFO - Updated checkpointInterval=60 on bucket 'default' on 172.23.105.151 [2021-05-11 18:27:15,560] - [task:6412] INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.105.151 -u Administrator -p password -b default -s _default -c _default -n 1000 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac True -st 0 -en 999 -o False -sd False [2021-05-11 18:27:18,931] - [xdcrnewbasetests:3364] INFO - Updating keys @ C1 [2021-05-11 18:27:48,932] - [task:6412] INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.105.151 -u Administrator -p password -b default -s _default -c _default -n 1000 -pc 0 -pu 30 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac True -st 0 -en 999 -o False -sd False [2021-05-11 18:27:51,745] - [xdcrnewbasetests:3374] INFO - Deleting keys @ C1 [2021-05-11 18:28:21,775] - [task:6412] INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.105.151 -u Administrator -p password -b default -s _default -c _default -n 1000 -pc 0 -pu 0 -pd 30 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac True -st 0 -en 999 -o False -sd False [2021-05-11 18:28:24,167] - [xdcrnewbasetests:3576] INFO - Merging keys for replication Replication C1:default -> C2:default [2021-05-11 18:28:24,187] - [xdcrnewbasetests:3514] INFO - src_kvstore has 0 valid and 0 deleted keys [2021-05-11 18:28:24,188] - [xdcrnewbasetests:3516] INFO - dest kvstore has 0 valid and 0 deleted keys [2021-05-11 18:28:24,194] - [xdcrnewbasetests:3566] INFO - After merging: destination bucket's kv_store now has 0 valid keys and 0 deleted keys [2021-05-11 18:28:24,281] - [data_helper:311] INFO - dict:{'ip': '172.23.105.151', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2021-05-11 18:28:24,282] - [data_helper:312] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:28:24,461] - [cluster_helper:371] INFO - Setting flush param on server {'ip': '172.23.105.151', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2021-05-11 18:28:24,461] - [mc_bin_client:680] INFO - setting param: exp_pager_stime 10 [2021-05-11 18:28:24,464] - [cluster_helper:385] INFO - Setting flush param on server {'ip': '172.23.105.151', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (2713022081, 0, b'') [2021-05-11 18:28:24,485] - [data_helper:311] INFO - dict:{'ip': '172.23.105.153', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2021-05-11 18:28:24,485] - [data_helper:312] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:28:24,552] - [cluster_helper:371] INFO - Setting flush param on server {'ip': '172.23.105.153', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2021-05-11 18:28:24,552] - [mc_bin_client:680] INFO - setting param: exp_pager_stime 10 [2021-05-11 18:28:24,553] - [cluster_helper:385] INFO - Setting flush param on server {'ip': '172.23.105.153', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (611820230, 0, b'') [2021-05-11 18:28:24,554] - [xdcrnewbasetests:1924] INFO - wait for expiry pager to run on all these nodes [2021-05-11 18:28:34,596] - [data_helper:311] INFO - dict:{'ip': '172.23.107.160', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2021-05-11 18:28:34,596] - [data_helper:312] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:28:34,649] - [cluster_helper:371] INFO - Setting flush param on server {'ip': '172.23.107.160', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2021-05-11 18:28:34,649] - [mc_bin_client:680] INFO - setting param: exp_pager_stime 10 [2021-05-11 18:28:34,650] - [cluster_helper:385] INFO - Setting flush param on server {'ip': '172.23.107.160', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (3303010757, 0, b'') [2021-05-11 18:28:34,671] - [data_helper:311] INFO - dict:{'ip': '172.23.107.184', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2021-05-11 18:28:34,671] - [data_helper:312] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:28:34,794] - [cluster_helper:371] INFO - Setting flush param on server {'ip': '172.23.107.184', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10 on default [2021-05-11 18:28:34,794] - [mc_bin_client:680] INFO - setting param: exp_pager_stime 10 [2021-05-11 18:28:34,795] - [cluster_helper:385] INFO - Setting flush param on server {'ip': '172.23.107.184', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 10, result: (1501794154, 0, b'') [2021-05-11 18:28:34,795] - [xdcrnewbasetests:1924] INFO - wait for expiry pager to run on all these nodes [2021-05-11 18:28:54,208] - [data_helper:314] INFO - creating direct client 172.23.105.151:11210 default [2021-05-11 18:28:54,284] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.151:8091',default bucket [2021-05-11 18:28:54,296] - [data_helper:314] INFO - creating direct client 172.23.105.153:11210 default [2021-05-11 18:28:54,382] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.153:8091',default bucket [2021-05-11 18:28:55,172] - [data_helper:314] INFO - creating direct client 172.23.107.160:11210 default [2021-05-11 18:28:55,323] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.160:8091',default bucket [2021-05-11 18:28:55,336] - [data_helper:314] INFO - creating direct client 172.23.107.184:11210 default [2021-05-11 18:28:55,442] - [task:956] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.184:8091',default bucket [2021-05-11 18:28:55,442] - [xdcrnewbasetests:2617] INFO - Waiting for dcp queue to drain on cluster node: 172.23.105.151 [2021-05-11 18:28:55,448] - [rest_client:2363] INFO - http://172.23.105.151:8091/pools/default/buckets/default/stats?zoom=minute [2021-05-11 18:28:56,243] - [xdcrnewbasetests:2627] INFO - Current dcp queue size on C1 for default is 0 [2021-05-11 18:28:56,243] - [xdcrnewbasetests:2617] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.160 [2021-05-11 18:28:56,248] - [rest_client:2363] INFO - http://172.23.107.160:8091/pools/default/buckets/default/stats?zoom=minute [2021-05-11 18:28:57,019] - [xdcrnewbasetests:2634] ERROR - 'ep_dcp_xdcr_items_remaining' [2021-05-11 18:28:57,273] - [xdcrnewbasetests:2657] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 0 [2021-05-11 18:28:57,336] - [xdcrnewbasetests:2654] WARNING - Stat "replication_changes_left" not found [2021-05-11 18:28:57,336] - [xdcrnewbasetests:3918] ERROR - local variable 'mutations' referenced before assignment [2021-05-11 18:28:57,360] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:28:57,490] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:28:57,887] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:28:58,248] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:28:58,250] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:28:58,353] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:28:58,652] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:28:58,964] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:28:59,941] - [task:2579] INFO - ---------------------------------------------------------------------------------------------------- [2021-05-11 18:28:59,941] - [task:2587] INFO - Item count on src:172.23.105.151 0 = 0 on dest:172.23.107.160 for bucket default src : scope "_default-> collection _default",dest: scope "_default-> collection _default" [2021-05-11 18:28:59,941] - [task:2596] INFO - ---------------------------------------------------------------------------------------------------- [2021-05-11 18:28:59,950] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 18:28:59,952] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:29:00,080] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:29:00,475] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:00,830] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:00,831] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:00,898] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:00,899] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:00,899] - [xdcrnewbasetests:464] INFO - [] [2021-05-11 18:29:00,899] - [xdcrnewbasetests:3648] INFO - Initial replication restart count on 172.23.105.151 :0, now :0 [2021-05-11 18:29:00,901] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:29:01,047] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:29:01,528] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:01,916] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:01,917] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:01,994] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:01,995] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:01,995] - [xdcrnewbasetests:464] INFO - [] [2021-05-11 18:29:01,995] - [xdcrnewbasetests:3648] INFO - Initial replication restart count on 172.23.105.153 :0, now :0 [2021-05-11 18:29:01,998] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:29:02,105] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:29:02,413] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:02,737] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:02,738] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:02,798] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:02,798] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:02,798] - [xdcrnewbasetests:464] INFO - [] [2021-05-11 18:29:02,798] - [xdcrnewbasetests:3648] INFO - Initial replication restart count on 172.23.107.160 :0, now :0 [2021-05-11 18:29:02,800] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:29:02,937] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:29:03,373] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:03,712] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:03,713] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:03,781] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:03,782] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:03,782] - [xdcrnewbasetests:464] INFO - [] [2021-05-11 18:29:03,782] - [xdcrnewbasetests:3648] INFO - Initial replication restart count on 172.23.107.184 :0, now :0 [2021-05-11 18:29:03,792] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 18:29:03,793] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:29:03,944] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:29:04,356] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:04,722] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:04,723] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:04,795] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:04,795] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:04,796] - [xdcrnewbasetests:3618] INFO - Initial panic: count on 172.23.105.151 :0, now :0 [2021-05-11 18:29:04,799] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:29:04,933] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:29:05,346] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:05,726] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:05,727] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:05,761] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:05,761] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:05,761] - [xdcrnewbasetests:3618] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.151 :0, now :0 [2021-05-11 18:29:05,763] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:29:05,900] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:29:06,360] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:06,737] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:06,738] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:06,811] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:06,811] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:06,811] - [xdcrnewbasetests:3618] INFO - Initial panic: count on 172.23.105.153 :0, now :0 [2021-05-11 18:29:06,813] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:29:06,957] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:29:07,385] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:07,770] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:07,771] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:07,843] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:07,843] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:07,843] - [xdcrnewbasetests:3618] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.105.153 :0, now :0 [2021-05-11 18:29:07,846] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:29:07,947] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:29:08,246] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:08,559] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:08,560] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:08,621] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:08,622] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:08,622] - [xdcrnewbasetests:3618] INFO - Initial panic: count on 172.23.107.160 :0, now :0 [2021-05-11 18:29:08,624] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:29:08,722] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:29:09,021] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:09,328] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:09,330] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:09,391] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:09,392] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:09,392] - [xdcrnewbasetests:3618] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.160 :0, now :0 [2021-05-11 18:29:09,394] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:29:09,523] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:29:09,942] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:10,282] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:10,283] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:10,351] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:10,352] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:10,353] - [xdcrnewbasetests:3618] INFO - Initial panic: count on 172.23.107.184 :0, now :0 [2021-05-11 18:29:10,354] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:29:10,500] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:29:10,911] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:29:11,250] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:29:11,250] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:29:11,278] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:29:11,278] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:29:11,279] - [xdcrnewbasetests:3618] INFO - Initial non-recoverable error from xmem client. response status=KEY_ENOENT count on 172.23.107.184 :0, now :0 [2021-05-11 18:29:11,318] - [xdcrnewbasetests:2799] INFO - ==== XDCRNewbasetests cleanup is started for test #16 load_with_ops ==== [2021-05-11 18:29:11,318] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 18:29:11,335] - [rest_client:1456] INFO - Deleting replication /controller/cancelXDCR/6278ccceb92e6cf9291193703f7681ec%2Fdefault%2Fdefault [2021-05-11 18:29:11,372] - [rest_client:1462] INFO - Replication deleted successfully [2021-05-11 18:29:11,379] - [rest_client:1399] INFO - removing remote cluster name:remote_cluster_C1-C2 [2021-05-11 18:29:11,425] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root] [2021-05-11 18:29:11,445] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.105.151 [2021-05-11 18:29:12,515] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:29:12,518] - [rest_client:137] INFO - node 172.23.105.151 existing buckets : [] [2021-05-11 18:29:12,518] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.105.151 [2021-05-11 18:29:12,523] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:29:12,534] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:29:12,537] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': 'ns_1@172.23.105.153', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:29:12,585] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:29:12,590] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:29:32,618] - [rest_client:1833] INFO - rebalance progress took 20.03 seconds [2021-05-11 18:29:32,618] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:29:42,640] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.105.151 port:8091 ssh_username:root ? [('ns_1@172.23.105.153', 8091)] [2021-05-11 18:29:42,645] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:29:42,645] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:29:42,649] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:29:42,658] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:29:42,662] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:29:42,677] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:29:42,677] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:29:42,681] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running Cluster instance shutdown with force [2021-05-11 18:29:42,681] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 18:29:42,704] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 18:29:42,721] - [bucket_helper:133] INFO - deleting existing buckets ['default'] on 172.23.107.160 [2021-05-11 18:29:44,043] - [bucket_helper:224] INFO - waiting for bucket deletion to complete.... [2021-05-11 18:29:44,047] - [rest_client:137] INFO - node 172.23.107.160 existing buckets : [] [2021-05-11 18:29:44,047] - [bucket_helper:156] INFO - deleted bucket : default from 172.23.107.160 [2021-05-11 18:29:44,054] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:29:44,066] - [cluster_helper:262] INFO - rebalancing all nodes in order to remove nodes [2021-05-11 18:29:44,068] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.160,ns_1@172.23.107.184', 'ejectedNodes': 'ns_1@172.23.107.184', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:29:44,107] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:29:44,110] - [rest_client:1894] INFO - rebalance percentage : 0.00 % [2021-05-11 18:30:04,141] - [rest_client:1833] INFO - rebalance progress took 20.03 seconds [2021-05-11 18:30:04,141] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:30:14,161] - [cluster_helper:325] INFO - removed all the nodes from cluster associated with ip:172.23.107.160 port:8091 ssh_username:root ? [('ns_1@172.23.107.184', 8091)] [2021-05-11 18:30:14,167] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:30:14,167] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:14,171] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:30:14,178] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:30:14,182] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:14,200] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:30:14,200] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:14,203] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running Cluster instance shutdown with force [2021-05-11 18:30:14,203] - [xdcrnewbasetests:2804] INFO - ==== XDCRNewbasetests cleanup is finished for test #16 load_with_ops === ok ---------------------------------------------------------------------- Ran 1 test in 318.753s OK suite_tearDown (xdcr.uniXDCR.unidirectional) ... Cluster instance shutdown with force summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0 summary so far suite xdcr.biXDCR.bidirectional , pass 0 , fail 1 failures so far... xdcr.biXDCR.bidirectional.load_with_async_ops summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 0 , fail 1 failures so far... rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops summary so far suite swaprebalance.SwapRebalanceBasicTests , pass 0 , fail 1 failures so far... swaprebalance.SwapRebalanceBasicTests.do_test summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 summary so far suite rackzone.rackzonetests.RackzoneTests , pass 2 , fail 0 summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_backup_restore_test.EnterpriseBackupRestoreTest , pass 1 , fail 0 summary so far suite security.x509tests.x509tests , pass 1 , fail 0 summary so far suite ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest , pass 1 , fail 0 summary so far suite clitest.importexporttest.ImportExportTests , pass 1 , fail 0 summary so far suite buckettests.CreateBucketTests , pass 1 , fail 0 summary so far suite tuqquery.n1ql_collections_end2end.QueryCollectionsEnd2EndTests , pass 1 , fail 0 testrunner logs, diags and results are available under /home/jenkins/workspace/build_sanity_matrix/DISTRO/centos7/TYPE/4node_1/logs/testrunner-21-May-11_17-05-17/test_16 *** Tests executed count: 16 Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_ops [2021-05-11 18:30:14,217] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 18:30:14,220] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:30:14,353] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:30:14,741] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:15,090] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:15,091] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:15,121] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:15,122] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:15,123] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:30:15,248] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:30:15,637] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:15,990] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:15,991] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:16,020] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:16,020] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:16,022] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:30:16,158] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:30:16,606] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:16,979] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:16,980] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:17,009] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:17,010] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:17,012] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:30:17,144] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:30:17,569] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:17,937] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:17,938] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:17,968] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:17,968] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:17,970] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:30:18,066] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:30:18,353] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:18,668] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:18,669] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:18,733] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:18,734] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:18,737] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:30:18,836] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:30:19,123] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:19,441] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:19,442] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:19,508] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:19,509] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:19,511] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:30:19,659] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:30:20,161] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:20,510] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:20,511] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "panic:" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:20,580] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:20,581] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:20,583] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:30:20,712] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:30:21,122] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:21,472] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:21,474] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:21,539] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:21,539] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:21,540] - [xdcrnewbasetests:2981] INFO - {'172.23.105.151': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.105.153': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.160': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}, '172.23.107.184': {'panic:': 0, 'non-recoverable error from xmem client. response status=KEY_ENOENT': 0}} [2021-05-11 18:30:21,550] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-05-11 18:30:21,551] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:30:21,678] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:30:22,056] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:22,401] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:22,402] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:22,469] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:22,470] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:22,473] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:30:22,605] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:30:23,040] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:23,411] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:23,413] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:23,445] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:23,445] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:23,448] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:30:23,546] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:30:23,821] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:24,130] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:24,131] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:24,197] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:24,198] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:24,199] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:30:24,328] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:30:24,791] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:25,132] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:25,133] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: zgrep "Try to fix Pipeline" /opt/couchbase/var/lib/couchbase/logs/goxdcr.log* [2021-05-11 18:30:25,159] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:25,159] - [xdcrnewbasetests:462] INFO - 0 [2021-05-11 18:30:25,159] - [xdcrnewbasetests:2999] INFO - {'172.23.105.151': 0, '172.23.105.153': 0, '172.23.107.160': 0, '172.23.107.184': 0} [2021-05-11 18:30:29,963] - [xdcrnewbasetests:2741] INFO - ==== XDCRNewbasetests setup is started for test #16 suite_tearDown ==== [2021-05-11 18:30:29,987] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 18:30:30,008] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.105.151 port:8091 ssh_username:root, ip:172.23.105.153 port:8091 ssh_username:root] [2021-05-11 18:30:30,015] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.151, nothing to delete [2021-05-11 18:30:30,019] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,032] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.151:8091 [2021-05-11 18:30:30,032] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,036] - [cluster_helper:86] INFO - ns_server @ 172.23.105.151:8091 is running [2021-05-11 18:30:30,049] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.105.153, nothing to delete [2021-05-11 18:30:30,054] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,078] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.105.153:8091 [2021-05-11 18:30:30,078] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,085] - [cluster_helper:86] INFO - ns_server @ 172.23.105.153:8091 is running [2021-05-11 18:30:30,085] - [xdcrnewbasetests:1347] INFO - removing xdcr/nodes settings [2021-05-11 18:30:30,104] - [xdcrnewbasetests:1357] INFO - cleanup [ip:172.23.107.160 port:8091 ssh_username:root, ip:172.23.107.184 port:8091 ssh_username:root] [2021-05-11 18:30:30,110] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.160, nothing to delete [2021-05-11 18:30:30,113] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,126] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.160:8091 [2021-05-11 18:30:30,126] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,130] - [cluster_helper:86] INFO - ns_server @ 172.23.107.160:8091 is running [2021-05-11 18:30:30,137] - [bucket_helper:158] INFO - Could not find any buckets for node 172.23.107.184, nothing to delete [2021-05-11 18:30:30,140] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,154] - [cluster_helper:82] INFO - waiting for ns_server @ 172.23.107.184:8091 [2021-05-11 18:30:30,154] - [rest_client:41] INFO - -->is_ns_server_running? [2021-05-11 18:30:30,157] - [cluster_helper:86] INFO - ns_server @ 172.23.107.184:8091 is running [2021-05-11 18:30:30,157] - [xdcrnewbasetests:3009] INFO - Initializing all clusters... [2021-05-11 18:30:30,164] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:30:30,295] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:30:30,701] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:31,051] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:31,063] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:30:31,124] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:31,125] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:30:31,125] - [xdcrnewbasetests:1239] INFO - Enabled diag/eval for non-local hosts from 172.23.105.151 [2021-05-11 18:30:31,128] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:30:31,973] - [task:152] INFO - server: ip:172.23.105.151 port:8091 ssh_username:root, nodes/self [2021-05-11 18:30:31,977] - [task:157] INFO - {'uptime': '392', 'memoryTotal': 8192827392, 'memoryFree': 6822309888, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.105.151:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4187, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.151', 'ip': '172.23.105.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7813, 'curr_items': 0} [2021-05-11 18:30:31,982] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:30:31,987] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:30:31,988] - [rest_client:1050] INFO - settings/web params on 172.23.105.151:8091:port=8091&username=Administrator&password=password [2021-05-11 18:30:32,068] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:30:32,070] - [remote_util:299] INFO - SSH Connecting to 172.23.105.151 with username:root, attempt#1 of 5 [2021-05-11 18:30:32,196] - [remote_util:335] INFO - SSH Connected to 172.23.105.151 as root [2021-05-11 18:30:32,583] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:32,933] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:32,946] - [remote_util:3427] INFO - running command.raw on 172.23.105.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:30:33,005] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:33,005] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:30:33,008] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:30:33,010] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.151:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:30:33,021] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:30:33,039] - [task:152] INFO - server: ip:172.23.105.153 port:8091 ssh_username:root, nodes/self [2021-05-11 18:30:33,043] - [task:157] INFO - {'uptime': '380', 'memoryTotal': 8191283200, 'memoryFree': 6963036160, 'mcdMemoryReserved': 6249, 'mcdMemoryAllocated': 6249, 'status': 'healthy', 'hostname': '172.23.105.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.153', 'ip': '172.23.105.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7811, 'curr_items': 0} [2021-05-11 18:30:33,045] - [rest_client:1022] ERROR - GET http://172.23.105.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:30:33,046] - [rest_client:1147] INFO - pools/default params : memoryQuota=4186 [2021-05-11 18:30:33,050] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:30:33,051] - [rest_client:1050] INFO - settings/web params on 172.23.105.153:8091:port=8091&username=Administrator&password=password [2021-05-11 18:30:33,150] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:30:33,152] - [remote_util:299] INFO - SSH Connecting to 172.23.105.153 with username:root, attempt#1 of 5 [2021-05-11 18:30:33,292] - [remote_util:335] INFO - SSH Connected to 172.23.105.153 as root [2021-05-11 18:30:33,739] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:30:34,103] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:30:34,116] - [remote_util:3427] INFO - running command.raw on 172.23.105.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:30:34,182] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:30:34,182] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:30:34,186] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:30:34,188] - [rest_client:1750] INFO - /diag/eval status on 172.23.105.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:30:34,195] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:30:35,215] - [task:769] INFO - adding node 172.23.105.153:8091 to cluster [2021-05-11 18:30:35,215] - [rest_client:1500] INFO - adding remote node @172.23.105.153:8091 to this cluster @172.23.105.151:8091 [2021-05-11 18:30:45,236] - [rest_client:1833] INFO - rebalance progress took 10.02 seconds [2021-05-11 18:30:45,236] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:31:04,035] - [rest_client:1727] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.151,ns_1@172.23.105.153', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:31:09,707] - [rest_client:1732] INFO - rebalance operation started [2021-05-11 18:31:19,716] - [task:839] INFO - Rebalance - status: none, progress: 100.00% [2021-05-11 18:31:19,722] - [task:898] INFO - rebalancing was completed with progress: 100% in 10.01476764678955 sec [2021-05-11 18:31:19,729] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:31:19,829] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:31:20,132] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:31:20,439] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:31:20,451] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:31:20,653] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:31:20,653] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:31:20,654] - [xdcrnewbasetests:1239] INFO - Enabled diag/eval for non-local hosts from 172.23.107.160 [2021-05-11 18:31:20,657] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:31:21,035] - [task:152] INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self [2021-05-11 18:31:21,039] - [task:157] INFO - {'uptime': '413', 'memoryTotal': 7934996480, 'memoryFree': 6771752960, 'mcdMemoryReserved': 6053, 'mcdMemoryAllocated': 6053, 'status': 'healthy', 'hostname': '172.23.107.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4055, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.160', 'ip': '172.23.107.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7567, 'curr_items': 0} [2021-05-11 18:31:21,044] - [rest_client:1147] INFO - pools/default params : memoryQuota=4055 [2021-05-11 18:31:21,049] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:31:21,049] - [rest_client:1050] INFO - settings/web params on 172.23.107.160:8091:port=8091&username=Administrator&password=password [2021-05-11 18:31:21,136] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:31:21,137] - [remote_util:299] INFO - SSH Connecting to 172.23.107.160 with username:root, attempt#1 of 5 [2021-05-11 18:31:21,233] - [remote_util:335] INFO - SSH Connected to 172.23.107.160 as root [2021-05-11 18:31:21,502] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:31:21,813] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:31:21,833] - [remote_util:3427] INFO - running command.raw on 172.23.107.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:31:22,024] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:31:22,025] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:31:22,029] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:31:22,032] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:31:22,037] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:31:22,057] - [task:152] INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self [2021-05-11 18:31:22,061] - [task:157] INFO - {'uptime': '397', 'memoryTotal': 8192131072, 'memoryFree': 6975229952, 'mcdMemoryReserved': 6250, 'mcdMemoryAllocated': 6250, 'status': 'healthy', 'hostname': '172.23.107.184:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-5148-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 4021, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.184', 'ip': '172.23.107.184', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 7812, 'curr_items': 0} [2021-05-11 18:31:22,063] - [rest_client:1022] ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-05-11 18:31:22,064] - [rest_client:1147] INFO - pools/default params : memoryQuota=4187 [2021-05-11 18:31:22,070] - [rest_client:1045] INFO - --> in init_cluster...Administrator,password,8091 [2021-05-11 18:31:22,070] - [rest_client:1050] INFO - settings/web params on 172.23.107.184:8091:port=8091&username=Administrator&password=password [2021-05-11 18:31:22,181] - [rest_client:1052] INFO - --> status:True [2021-05-11 18:31:22,183] - [remote_util:299] INFO - SSH Connecting to 172.23.107.184 with username:root, attempt#1 of 5 [2021-05-11 18:31:22,321] - [remote_util:335] INFO - SSH Connected to 172.23.107.184 as root [2021-05-11 18:31:22,769] - [remote_util:3587] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-05-11 18:31:23,119] - [remote_util:3743] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-05-11 18:31:23,130] - [remote_util:3427] INFO - running command.raw on 172.23.107.184: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-05-11 18:31:23,193] - [remote_util:3475] INFO - command executed successfully with root [2021-05-11 18:31:23,193] - [remote_util:5220] INFO - ['ok'] [2021-05-11 18:31:23,196] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:31:23,198] - [rest_client:1750] INFO - /diag/eval status on 172.23.107.184:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-05-11 18:31:23,203] - [rest_client:1182] INFO - settings/indexes params : storageMode=plasma [2021-05-11 18:31:24,219] - [task:769] INFO - adding node 172.23.107.184:8091 to cluster [2021-05-11 18:31:24,219] - [rest_client:1500] INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.107.160:8091 [2021-05-11 18:31:34,228] - [rest_client:1833] INFO - rebalance progress took 10.01 seconds [2021-05-11 18:31:34,228] - [rest_client:1834] INFO - sleep for 10 seconds after rebalance... [2021-05-11 18:31:53,065] - [rest_client:1022] ERROR - GET http://172.23.107.160:8091/nodeStatuses body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password [2021-05-11 18:31:53,070] - [rest_client:1027] ERROR - socket error while connecting to http://172.23.107.160:8091/pools/default error [Errno 111] Connection refused [2021-05-11 18:31:56,080] - [rest_client:1727] INFO - rebalance params : {'knownNodes': '', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-05-11 18:31:56,085] - [rest_client:1022] ERROR - POST http://172.23.107.160:8091/controller/rebalance body: knownNodes=&ejectedNodes=&user=Administrator&password=password headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"empty_known_nodes":1}' auth: Administrator:password [2021-05-11 18:31:56,086] - [rest_client:1734] ERROR - rebalance operation failed: b'{"empty_known_nodes":1}' *** TestRunner *** + RESULT=0 + '[' centos7-4node_1 = centos7-4node ']' + '[' centos7-4node_1 = centos7-4node_1 ']' ++ curl http://dbapi.build.couchbase.com:8000/v1/builds/couchbase-server-7.0.0-5148/metadata % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 127 100 127 0 0 14577 0 --:--:-- --:--:-- --:--:-- 15875 + METADATA='{"data": {"unit_test": "fail", "jira_comments": true, "blackduck_scan": true, "builds_complete": "complete", "sanity": "pass"}}' + PREV_RESULT=echo + '{"data":' '{"unit_test":' '"fail",' '"jira_comments":' true, '"blackduck_scan":' true, '"builds_complete":' '"complete",' '"sanity":' '"pass"}}' + grep -E fail -c /tmp/jenkins6239551877371304978.sh: line 122: {"data":: command not found 0 + RESULT=0 + '[' 0 -eq 0 ']' + curl -d '{"sanity": "pass"}' -X POST -H 'Content-Type: application/json' http://dbapi.build.couchbase.com:8000/v1/builds/couchbase-server-7.0.0-5148/metadata % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 34 100 16 100 18 1733 1950 --:--:-- --:--:-- --:--:-- 2000 {"status": "ok"}+ echo '{"vms":' '["172.23.105.151",' '"172.23.105.153",' '"172.23.107.160",' '"172.23.107.184"]}' + curl -s -d @data.json -X DELETE -H 'Content-Type: application/json' http://dbapi.build.couchbase.com:8000/v1/reservations {"status": "ok"}+ rm -f data.json + rm -f conf/py-multi-node-sanity-1.conf + rm -f conf/py-multi-node-sanity-2.conf New run name is '#9106--7.0.0-5148' Archiving artifacts Recording test results [Checks API] No suitable checks publisher found. Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: '172.23.105.151 172.23.105.153 172.23.107.160 172.23.107.184' Sending e-mails to: wayne@couchbase.com Ritesh.Agarwal@couchbase.com ritam@couchbase.com Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: build-sanity-notify@couchbase.com Finished: UNSTABLE