Started by remote host 172.23.107.166 [EnvInject] - Loading node environment variables. Building remotely on slv-sc2404-32g-12c (P0 jython_slave) in workspace /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers] $ /bin/sh -xe /tmp/jenkins1154085761116633940.sh ++ echo eventing-rebalance_bucket_op_with_timers-Feb-14-21:07:13-7.0.0-4454 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='eventing rebalance_bucket_op_with_timers' + echo Desc: 7.0.0-4454 - eventing rebalance_bucket_op_with_timers - centos Desc: 7.0.0-4454 - eventing rebalance_bucket_op_with_timers - centos + echo newState=available + newState=available Success build forhudson.tasks.Shell@3bc01fe9 [description-setter] Description set: 7.0.0-4454 - eventing rebalance_bucket_op_with_timers - centos Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@3d0c3035 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@4994af6d Cloning the remote Git repository Using shallow clone Cloning repository git://github.com/couchbase/testrunner > /usr/bin/git init /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbase/testrunner # timeout=10 Fetching upstream changes from git://github.com/couchbase/testrunner > /usr/bin/git fetch --tags --progress git://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 980880195f90dc6856c8be70ed976d5e8fdd4252 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 980880195f90dc6856c8be70ed976d5e8fdd4252 > /usr/bin/git rev-list 980880195f90dc6856c8be70ed976d5e8fdd4252 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #314953 jenkins-test_suite_executor-314953 # timeout=10 No emails were triggered. [centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers] $ /bin/sh -xe /tmp/jenkins5556143862602627623.sh + echo Desc: eventing-rebalance_bucket_op_with_timers-Feb-14-21:07:13-7.0.0-4454 Desc: eventing-rebalance_bucket_op_with_timers-Feb-14-21:07:13-7.0.0-4454 [description-setter] Could not determine description. [centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers] $ /bin/sh -xe /tmp/jenkins2621381146034770221.sh [centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers] $ /bin/sh -xe /tmp/jenkins7230591054452821377.sh + py_executable=python2 + echo 7.0.0-4454 + grep '7\.0' 7.0.0-4454 + py_executable=python3 + [[ 7.0.0-4454 > 6.5 ]] + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + rerun_job=true + touch rerun_props_file + '[' false == false ']' + '[' true == true ']' + python3 scripts/rerun_jobs.py 7.0.0-4454 --executor_jenkins_job --manual_run [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers] $ /bin/bash /tmp/jenkins7147452810168804285.sh 7.0.0-4454 Already on 'master' From git://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. Set ALLOW_HTP to False so test could run. HEAD is now at 9808801 eventing failover test issues the major release is 7 "172.23.104.237","172.23.104.243","172.23.104.245","172.23.104.246","172.23.104.247","172.23.104.248","172.23.104.249","172.23.106.6" Searching for httplib2 Best match: httplib2 0.17.0 Adding httplib2 0.17.0 to easy-install.pth file Using /usr/local/lib/python3.7/site-packages Processing dependencies for httplib2 Finished processing dependencies for httplib2 centos Loaded plugins: fastestmirror Determining fastest mirrors * base: mirrors.sonic.net * extras: repos.lax.quadranet.com * updates: mirrors.sonic.net Package 2:docker-1.13.1-203.git0be3e21.el7.centos.x86_64 already installed and latest version Nothing to do Using default tag: latest Trying to pull repository docker.io/jamesdbloom/mockserver ... latest: Pulling from docker.io/jamesdbloom/mockserver Digest: sha256:bc41010b9e920f5e1b75a226e8eda05eb4eb9843dad042e9e3950d1b7de823f4 Status: Image is up to date for docker.io/jamesdbloom/mockserver:latest [global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [_1] ip:dynamic port:8091 n1ql_port:18093 index_port:9102 services:kv,index,n1ql [_2] ip:dynamic port:8091 [_3] ip:dynamic port:8091 [_4] ip:dynamic port:8091 [_5] ip:dynamic port:8091 [_6] ip:dynamic port:8091 [_7] ip:dynamic port:8091 [_8] ip:dynamic port:8091 python3 scripts/populateIni.py -s "172.23.104.237","172.23.104.243","172.23.104.245","172.23.104.246","172.23.104.247","172.23.104.248","172.23.104.249","172.23.106.6" -d None -a None -i /tmp/testexec_reformat.110140.ini -p centos -o /tmp/testexec.110140.ini -k {} INFO:root:SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.104.246 as root INFO:root:SSH Connected to 172.23.104.248 as root INFO:root:SSH Connected to 172.23.104.243 as root INFO:root:SSH Connected to 172.23.104.237 as root INFO:root:SSH Connected to 172.23.104.247 as root INFO:root:SSH Connected to 172.23.104.249 as root INFO:root:SSH Connected to 172.23.104.245 as root INFO:root:SSH Connected to 172.23.106.6 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.246: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.248: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.245: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.237: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.6: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.247: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.249: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.243: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root in main the ini file is /tmp/testexec_reformat.110140.ini the given server info is "172.23.104.237","172.23.104.243","172.23.104.245","172.23.104.246","172.23.104.247","172.23.104.248","172.23.104.249","172.23.106.6" Collecting memory info from 172.23.104.246 Collecting memory info from 172.23.104.248 Collecting memory info from 172.23.104.245 Collecting memory info from 172.23.104.237 Collecting memory info from 172.23.106.6 Collecting memory info from 172.23.104.247 Collecting memory info from 172.23.104.249 Collecting memory info from 172.23.104.243 the servers memory info is [('172.23.104.246', 3618892), ('172.23.104.248', 3618892), ('172.23.106.6', 4103360), ('172.23.104.237', 4103368), ('172.23.104.247', 4120548), ('172.23.104.249', 4120552), ('172.23.104.243', 4120552), ('172.23.104.245', 24496164)] [global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [_1] ip:172.23.104.246 port:8091 n1ql_port:18093 index_port:9102 services:kv,index,n1ql [_2] ip:172.23.104.248 port:8091 [_3] ip:172.23.106.6 port:8091 [_4] ip:172.23.104.237 port:8091 [_5] ip:172.23.104.247 port:8091 [_6] ip:172.23.104.249 port:8091 [_7] ip:172.23.104.243 port:8091 [_8] ip:172.23.104.245 port:8091 extra install is Local time: Sun 2021-02-14 21:07:25 PST Universal time: Mon 2021-02-15 05:07:25 UTC RTC time: Mon 2021-02-15 05:07:25 Time zone: America/Los_Angeles (PST, -0800) NTP enabled: no NTP synchronized: yes RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2020-11-01 01:59:59 PDT Sun 2020-11-01 01:00:00 PST Next DST change: DST begins (the clock jumps one hour forward) at Sun 2021-03-14 01:59:59 PST Sun 2021-03-14 03:00:00 PDT python3 scripts/ssh.py -i /tmp/testexec_root.110140.ini iptables -F INFO:root:SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.104.248 as root INFO:root:SSH Connected to 172.23.104.246 as root INFO:root:SSH Connected to 172.23.104.245 as root INFO:root:SSH Connected to 172.23.104.249 as root INFO:root:SSH Connected to 172.23.104.243 as root INFO:root:SSH Connected to 172.23.106.6 as root INFO:root:SSH Connected to 172.23.104.237 as root INFO:root:SSH Connected to 172.23.104.247 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.246: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.248: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.6: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.249: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.237: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.247: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.245: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.243: iptables -F INFO:root:command executed successfully with root 172.23.104.246 172.23.104.248 172.23.106.6 172.23.104.245 172.23.104.249 172.23.104.237 172.23.104.247 172.23.104.243 python3 scripts/new_install.py -i /tmp/testexec.110140.ini -p timeout=1800,skip_local_download=False,get-cbcollect-info=True,version=7.0.0-4454,product=cb,debug_logs=True,ntp=True,url= 2021-02-14 21:07:26,951 - root - WARNING - URL: is not valid, will use version to locate build 2021-02-14 21:07:26,952 - root - INFO - SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:07:27,037 - root - INFO - SSH Connected to 172.23.104.246 as root 2021-02-14 21:07:27,202 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:27,493 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:27,494 - root - INFO - SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:07:27,579 - root - INFO - SSH Connected to 172.23.104.248 as root 2021-02-14 21:07:27,749 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:28,036 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:28,038 - root - INFO - SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:07:28,137 - root - INFO - SSH Connected to 172.23.106.6 as root 2021-02-14 21:07:28,379 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:28,677 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:28,679 - root - INFO - SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:07:28,772 - root - INFO - SSH Connected to 172.23.104.237 as root 2021-02-14 21:07:29,038 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:29,321 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:29,323 - root - INFO - SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:07:29,419 - root - INFO - SSH Connected to 172.23.104.247 as root 2021-02-14 21:07:29,693 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:29,980 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:29,981 - root - INFO - SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:07:30,074 - root - INFO - SSH Connected to 172.23.104.249 as root 2021-02-14 21:07:30,330 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:30,616 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:30,618 - root - INFO - SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:07:30,718 - root - INFO - SSH Connected to 172.23.104.243 as root 2021-02-14 21:07:31,026 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:31,317 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:31,318 - root - INFO - SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:07:31,416 - root - INFO - SSH Connected to 172.23.104.245 as root 2021-02-14 21:07:31,680 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:31,970 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:31,972 - root - INFO - SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:07:32,053 - root - INFO - SSH Connected to 172.23.104.246 as root 2021-02-14 21:07:32,210 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:32,494 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:32,496 - root - INFO - SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:07:32,580 - root - INFO - SSH Connected to 172.23.104.248 as root 2021-02-14 21:07:32,741 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:33,028 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:33,029 - root - INFO - SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:07:33,132 - root - INFO - SSH Connected to 172.23.106.6 as root 2021-02-14 21:07:33,392 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:33,694 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:33,696 - root - INFO - SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:07:33,790 - root - INFO - SSH Connected to 172.23.104.237 as root 2021-02-14 21:07:34,054 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:34,345 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:34,347 - root - INFO - SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:07:34,443 - root - INFO - SSH Connected to 172.23.104.247 as root 2021-02-14 21:07:34,717 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:35,003 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:35,005 - root - INFO - SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:07:35,101 - root - INFO - SSH Connected to 172.23.104.249 as root 2021-02-14 21:07:35,357 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:35,648 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:35,649 - root - INFO - SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:07:35,747 - root - INFO - SSH Connected to 172.23.104.243 as root 2021-02-14 21:07:36,053 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:36,346 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:36,348 - root - INFO - SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:07:36,445 - root - INFO - SSH Connected to 172.23.104.245 as root 2021-02-14 21:07:36,710 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:07:36,995 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:07:36,996 - root - INFO - Check if ntp is installed 2021-02-14 21:07:36,996 - root - INFO - This OS version centos 7 2021-02-14 21:07:36,996 - root - INFO - running command.raw on 172.23.104.246: systemctl status ntpd 2021-02-14 21:07:36,997 - root - INFO - Check if ntp is installed 2021-02-14 21:07:36,997 - root - INFO - This OS version centos 7 2021-02-14 21:07:36,997 - root - INFO - running command.raw on 172.23.104.248: systemctl status ntpd 2021-02-14 21:07:36,997 - root - INFO - Check if ntp is installed 2021-02-14 21:07:36,998 - root - INFO - This OS version centos 7 2021-02-14 21:07:36,998 - root - INFO - running command.raw on 172.23.106.6: systemctl status ntpd 2021-02-14 21:07:36,999 - root - INFO - Check if ntp is installed 2021-02-14 21:07:36,999 - root - INFO - This OS version centos 7 2021-02-14 21:07:36,999 - root - INFO - running command.raw on 172.23.104.237: systemctl status ntpd 2021-02-14 21:07:37,000 - root - INFO - Check if ntp is installed 2021-02-14 21:07:37,000 - root - INFO - This OS version centos 7 2021-02-14 21:07:37,000 - root - INFO - running command.raw on 172.23.104.247: systemctl status ntpd 2021-02-14 21:07:37,003 - root - INFO - Check if ntp is installed 2021-02-14 21:07:37,003 - root - INFO - This OS version centos 7 2021-02-14 21:07:37,003 - root - INFO - running command.raw on 172.23.104.249: systemctl status ntpd 2021-02-14 21:07:37,005 - root - INFO - Check if ntp is installed 2021-02-14 21:07:37,005 - root - INFO - This OS version centos 7 2021-02-14 21:07:37,005 - root - INFO - running command.raw on 172.23.104.243: systemctl status ntpd 2021-02-14 21:07:37,007 - root - INFO - Check if ntp is installed 2021-02-14 21:07:37,007 - root - INFO - This OS version centos 7 2021-02-14 21:07:37,007 - root - INFO - running command.raw on 172.23.104.245: systemctl status ntpd 2021-02-14 21:07:37,026 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,026 - root - INFO - running command.raw on 172.23.104.237: ntpstat 2021-02-14 21:07:37,028 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,028 - root - INFO - running command.raw on 172.23.106.6: ntpstat 2021-02-14 21:07:37,030 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,030 - root - INFO - running command.raw on 172.23.104.247: ntpstat 2021-02-14 21:07:37,032 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,032 - root - INFO - running command.raw on 172.23.104.249: ntpstat 2021-02-14 21:07:37,033 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,033 - root - INFO - running command.raw on 172.23.104.243: ntpstat 2021-02-14 21:07:37,034 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,035 - root - INFO - running command.raw on 172.23.104.246: ntpstat 2021-02-14 21:07:37,056 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,056 - root - INFO - running command.raw on 172.23.104.245: ntpstat 2021-02-14 21:07:37,060 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,060 - root - INFO - running command.raw on 172.23.104.248: ntpstat 2021-02-14 21:07:37,082 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,082 - root - INFO - running command.raw on 172.23.104.248: timedatectl status 2021-02-14 21:07:37,085 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,085 - root - INFO - running command.raw on 172.23.104.243: timedatectl status 2021-02-14 21:07:37,087 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,088 - root - INFO - running command.raw on 172.23.104.246: timedatectl status 2021-02-14 21:07:37,117 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,117 - root - INFO - running command.raw on 172.23.104.237: timedatectl status 2021-02-14 21:07:37,118 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,118 - root - INFO - running command.raw on 172.23.104.249: timedatectl status 2021-02-14 21:07:37,119 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,119 - root - INFO - running command.raw on 172.23.104.247: timedatectl status 2021-02-14 21:07:37,128 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,128 - root - INFO - running command.raw on 172.23.106.6: timedatectl status 2021-02-14 21:07:37,144 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,144 - root - INFO - running command.raw on 172.23.104.245: timedatectl status 2021-02-14 21:07:37,161 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,161 - root - INFO - running command.raw on 172.23.104.249: date 2021-02-14 21:07:37,162 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,163 - root - INFO - running command.raw on 172.23.104.247: date 2021-02-14 21:07:37,165 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,165 - root - INFO - running command.raw on 172.23.104.246: date 2021-02-14 21:07:37,167 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,168 - root - INFO - running command.raw on 172.23.104.243: date 2021-02-14 21:07:37,176 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,176 - root - INFO - running command.raw on 172.23.106.6: date 2021-02-14 21:07:37,191 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,191 - root - INFO - running command.raw on 172.23.104.248: date 2021-02-14 21:07:37,200 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,200 - root - INFO - running command.raw on 172.23.104.237: date 2021-02-14 21:07:37,214 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,215 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,218 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,219 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,221 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,221 - root - INFO - running command.raw on 172.23.104.245: date 2021-02-14 21:07:37,229 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,244 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,253 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,266 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.249 2021-02-14 21:07:37,268 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.247 2021-02-14 21:07:37,269 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.246 2021-02-14 21:07:37,271 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.243 2021-02-14 21:07:37,273 - root - INFO - command executed successfully with root 2021-02-14 21:07:37,281 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.106.6 2021-02-14 21:07:37,296 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.248 2021-02-14 21:07:37,304 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.237 2021-02-14 21:07:37,324 - root - INFO - ['Sun Feb 14 21:07:37 PST 2021'] IP: 172.23.104.245 2021-02-14 21:07:37,324 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,327 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,327 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,329 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,329 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,330 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,330 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,331 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,332 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,333 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,333 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,334 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,334 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,339 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,339 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:07:37,340 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm is live 2021-02-14 21:07:37,341 - root - INFO - Downloading build binary to /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm.. 2021-02-14 21:07:38,780 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.246 2021-02-14 21:07:38,781 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.248 2021-02-14 21:07:38,783 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.106.6 2021-02-14 21:07:38,783 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.237 2021-02-14 21:07:38,784 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.247 2021-02-14 21:07:38,786 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.249 2021-02-14 21:07:38,787 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.243 2021-02-14 21:07:38,788 - root - INFO - Copying /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm to 172.23.104.245 2021-02-14 21:10:10,926 - root - INFO - Done copying build to 172.23.104.237. 2021-02-14 21:10:11,528 - root - INFO - Done copying build to 172.23.104.246. 2021-02-14 21:10:11,907 - root - INFO - Done copying build to 172.23.104.243. 2021-02-14 21:10:11,947 - root - INFO - Done copying build to 172.23.104.247. 2021-02-14 21:10:12,115 - root - INFO - Done copying build to 172.23.104.249. 2021-02-14 21:10:12,219 - root - INFO - Done copying build to 172.23.104.248. 2021-02-14 21:10:12,392 - root - INFO - Done copying build to 172.23.106.6. 2021-02-14 21:10:12,972 - root - INFO - Done copying build to 172.23.104.245. 2021-02-14 21:10:12,972 - root - INFO - running command.raw on 172.23.104.246: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:12,992 - root - INFO - command executed successfully with root 2021-02-14 21:10:12,992 - root - INFO - running command.raw on 172.23.104.246: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,049 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,049 - root - INFO - running command.raw on 172.23.104.246: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,100 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,101 - root - INFO - running command.raw on 172.23.104.248: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,119 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,120 - root - INFO - running command.raw on 172.23.104.248: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,180 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,180 - root - INFO - running command.raw on 172.23.104.248: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,233 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,233 - root - INFO - running command.raw on 172.23.106.6: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,254 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,254 - root - INFO - running command.raw on 172.23.106.6: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,313 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,314 - root - INFO - running command.raw on 172.23.106.6: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,368 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,368 - root - INFO - running command.raw on 172.23.104.237: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,387 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,387 - root - INFO - running command.raw on 172.23.104.237: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,444 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,444 - root - INFO - running command.raw on 172.23.104.237: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,496 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,496 - root - INFO - running command.raw on 172.23.104.247: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,515 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,515 - root - INFO - running command.raw on 172.23.104.247: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,573 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,574 - root - INFO - running command.raw on 172.23.104.247: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,624 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,625 - root - INFO - running command.raw on 172.23.104.249: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,642 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,642 - root - INFO - running command.raw on 172.23.104.249: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,702 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,703 - root - INFO - running command.raw on 172.23.104.249: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,755 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,755 - root - INFO - running command.raw on 172.23.104.243: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,774 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,774 - root - INFO - running command.raw on 172.23.104.243: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,833 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,833 - root - INFO - running command.raw on 172.23.104.243: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,885 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,885 - root - INFO - running command.raw on 172.23.104.245: ls -lh /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:13,905 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,905 - root - INFO - running command.raw on 172.23.104.245: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4454/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-02-14 21:10:13,969 - root - INFO - command executed successfully with root 2021-02-14 21:10:13,969 - root - INFO - running command.raw on 172.23.104.245: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm 2021-02-14 21:10:14,021 - root - INFO - command executed successfully with root 2021-02-14 21:10:14,021 - root - INFO - running command.raw on 172.23.104.246: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,022 - root - INFO - running command.raw on 172.23.104.248: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,023 - root - INFO - running command.raw on 172.23.106.6: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,024 - root - INFO - running command.raw on 172.23.104.237: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,024 - root - INFO - running command.raw on 172.23.104.247: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,025 - root - INFO - running command.raw on 172.23.104.249: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,026 - root - INFO - running command.raw on 172.23.104.243: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:14,028 - root - INFO - running command.raw on 172.23.104.245: systemctl stop couchbase-server; rpm -e couchbase-server; rm -rf /opt/couchbase; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0 2021-02-14 21:10:16,371 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:16,572 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:16,843 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:16,853 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:16,959 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:17,137 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:17,210 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:17,309 - root - INFO - Done with uninstall on 172.23.104.248. 2021-02-14 21:10:17,310 - root - INFO - running command.raw on 172.23.104.248: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:17,439 - root - INFO - command executed with root but got an error ['warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave', 'warning: file /opt/couchbase/var/lib/couchbase/config/dist_cfg: remove failed: No such file or directory', 'warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/ ... 2021-02-14 21:10:17,504 - root - INFO - Done with uninstall on 172.23.104.246. 2021-02-14 21:10:17,504 - root - INFO - running command.raw on 172.23.104.246: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:17,864 - root - INFO - Done with uninstall on 172.23.104.237. 2021-02-14 21:10:17,864 - root - INFO - running command.raw on 172.23.104.237: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:17,899 - root - INFO - Done with uninstall on 172.23.104.247. 2021-02-14 21:10:17,899 - root - INFO - running command.raw on 172.23.104.247: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:17,936 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:17,936 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.248.. 2021-02-14 21:10:17,991 - root - INFO - Done with uninstall on 172.23.104.249. 2021-02-14 21:10:17,991 - root - INFO - running command.raw on 172.23.104.249: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:18,102 - root - INFO - Done with uninstall on 172.23.104.245. 2021-02-14 21:10:18,102 - root - INFO - running command.raw on 172.23.104.245: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:18,162 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:18,162 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.246.. 2021-02-14 21:10:18,201 - root - INFO - Done with uninstall on 172.23.104.243. 2021-02-14 21:10:18,202 - root - INFO - running command.raw on 172.23.104.243: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:18,446 - root - INFO - Done with uninstall on 172.23.106.6. 2021-02-14 21:10:18,446 - root - INFO - running command.raw on 172.23.106.6: yes | yum remove `rpm -qa | grep couchbase` 2021-02-14 21:10:18,879 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:18,879 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.237.. 2021-02-14 21:10:18,949 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:18,950 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.247.. 2021-02-14 21:10:18,975 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:18,975 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.249.. 2021-02-14 21:10:19,235 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:19,235 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.243.. 2021-02-14 21:10:19,391 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:19,391 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.245.. 2021-02-14 21:10:19,696 - root - INFO - command executed with root but got an error ['Error: Need to pass a list of pkgs to remove', ' Mini usage:', '', 'erase PACKAGE...', '', 'Remove a package or packages from your system', '', 'aliases: remove, autoremove, erase-n, erase-na, erase-nevra, autoremove-n, autoremove-na, autoremove-nevra, remove-n, remove-na, remove-nevra'] ... 2021-02-14 21:10:19,696 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.106.6.. 2021-02-14 21:10:37,956 - root - INFO - running command.raw on 172.23.104.248: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:37,972 - root - INFO - command executed successfully with root 2021-02-14 21:10:37,973 - root - INFO - running command.raw on 172.23.104.248: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:38,182 - root - INFO - running command.raw on 172.23.104.246: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:38,200 - root - INFO - command executed successfully with root 2021-02-14 21:10:38,200 - root - INFO - running command.raw on 172.23.104.246: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:38,900 - root - INFO - running command.raw on 172.23.104.237: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:38,915 - root - INFO - command executed successfully with root 2021-02-14 21:10:38,915 - root - INFO - running command.raw on 172.23.104.237: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:38,963 - root - INFO - running command.raw on 172.23.104.247: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:38,978 - root - INFO - command executed successfully with root 2021-02-14 21:10:38,978 - root - INFO - running command.raw on 172.23.104.247: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:38,995 - root - INFO - running command.raw on 172.23.104.249: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:39,010 - root - INFO - command executed successfully with root 2021-02-14 21:10:39,010 - root - INFO - running command.raw on 172.23.104.249: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:39,255 - root - INFO - running command.raw on 172.23.104.243: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:39,271 - root - INFO - command executed successfully with root 2021-02-14 21:10:39,271 - root - INFO - running command.raw on 172.23.104.243: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:39,411 - root - INFO - running command.raw on 172.23.104.245: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:39,428 - root - INFO - command executed successfully with root 2021-02-14 21:10:39,429 - root - INFO - running command.raw on 172.23.104.245: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:10:39,717 - root - INFO - running command.raw on 172.23.106.6: /sbin/sysctl vm.swappiness=0; echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag; 2021-02-14 21:10:39,734 - root - INFO - command executed successfully with root 2021-02-14 21:10:39,734 - root - INFO - running command.raw on 172.23.106.6: yes | yum localinstall -y /tmp/couchbase-server-enterprise-7.0.0-4454-centos7.x86_64.rpm > /dev/null && echo 1 || echo 0 2021-02-14 21:11:32,845 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:32,845 - root - INFO - running command.raw on 172.23.104.248: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:32,908 - root - INFO - command executed successfully with root 2021-02-14 21:11:32,908 - root - INFO - Done with install on 172.23.104.248. 2021-02-14 21:11:32,908 - root - INFO - Waiting 60s for 172.23.104.248 to be initialized.. 2021-02-14 21:11:33,196 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:33,196 - root - INFO - running command.raw on 172.23.104.246: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:33,258 - root - INFO - command executed successfully with root 2021-02-14 21:11:33,258 - root - INFO - Done with install on 172.23.104.246. 2021-02-14 21:11:33,258 - root - INFO - Waiting 60s for 172.23.104.246 to be initialized.. 2021-02-14 21:11:35,733 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:35,734 - root - INFO - running command.raw on 172.23.104.249: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:35,753 - root - INFO - command executed successfully with root 2021-02-14 21:11:35,753 - root - INFO - Done with install on 172.23.104.249. 2021-02-14 21:11:35,753 - root - INFO - Waiting 60s for 172.23.104.249 to be initialized.. 2021-02-14 21:11:37,014 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:37,014 - root - INFO - running command.raw on 172.23.104.237: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:37,075 - root - INFO - command executed successfully with root 2021-02-14 21:11:37,075 - root - INFO - Done with install on 172.23.104.237. 2021-02-14 21:11:37,076 - root - INFO - Waiting 60s for 172.23.104.237 to be initialized.. 2021-02-14 21:11:39,839 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:39,839 - root - INFO - running command.raw on 172.23.104.245: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:39,901 - root - INFO - command executed successfully with root 2021-02-14 21:11:39,901 - root - INFO - Done with install on 172.23.104.245. 2021-02-14 21:11:39,901 - root - INFO - Waiting 60s for 172.23.104.245 to be initialized.. 2021-02-14 21:11:40,656 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:40,656 - root - INFO - running command.raw on 172.23.104.243: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:40,722 - root - INFO - command executed successfully with root 2021-02-14 21:11:40,722 - root - INFO - Done with install on 172.23.104.243. 2021-02-14 21:11:40,722 - root - INFO - Waiting 60s for 172.23.104.243 to be initialized.. 2021-02-14 21:11:40,875 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:40,875 - root - INFO - running command.raw on 172.23.104.247: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:40,901 - root - INFO - command executed successfully with root 2021-02-14 21:11:40,901 - root - INFO - Done with install on 172.23.104.247. 2021-02-14 21:11:40,901 - root - INFO - Waiting 60s for 172.23.104.247 to be initialized.. 2021-02-14 21:11:44,881 - root - INFO - command executed with root but got an error ['Warning: RPMDB altered outside of yum.'] ... 2021-02-14 21:11:44,882 - root - INFO - running command.raw on 172.23.106.6: systemctl -q is-active couchbase-server && echo 1 || echo 0 2021-02-14 21:11:44,906 - root - INFO - command executed successfully with root 2021-02-14 21:11:44,907 - root - INFO - Done with install on 172.23.106.6. 2021-02-14 21:11:44,907 - root - INFO - Waiting 60s for 172.23.106.6 to be initialized.. 2021-02-14 21:12:32,915 - root - INFO - running command.raw on 172.23.104.248: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.248 -u Administrator -p password 2021-02-14 21:12:33,119 - root - INFO - command executed successfully with root 2021-02-14 21:12:33,258 - root - INFO - running command.raw on 172.23.104.246: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.246 -u Administrator -p password 2021-02-14 21:12:33,479 - root - INFO - command executed successfully with root 2021-02-14 21:12:34,131 - root - INFO - Setting KV memory quota as 1894 MB on 172.23.104.248 2021-02-14 21:12:34,131 - root - INFO - pools/default params : memoryQuota=1894 2021-02-14 21:12:34,138 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:34,138 - root - INFO - /node/controller/setupServices params on 172.23.104.248: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:34,166 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:34,166 - root - INFO - settings/web params on 172.23.104.248:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:34,216 - root - INFO - --> status:True 2021-02-14 21:12:34,216 - root - INFO - Done with init on 172.23.104.248. 2021-02-14 21:12:34,216 - root - INFO - running command.raw on 172.23.104.248: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:34,235 - root - INFO - command executed successfully with root 2021-02-14 21:12:34,235 - root - INFO - Done with cleanup on 172.23.104.248. 2021-02-14 21:12:34,489 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.104.246 2021-02-14 21:12:34,490 - root - INFO - pools/default params : indexMemoryQuota=256 2021-02-14 21:12:34,496 - root - INFO - Setting KV memory quota as 1638 MB on 172.23.104.246 2021-02-14 21:12:34,496 - root - INFO - pools/default params : memoryQuota=1638 2021-02-14 21:12:34,500 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql']) 2021-02-14 21:12:34,500 - root - INFO - /node/controller/setupServices params on 172.23.104.246: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql 2021-02-14 21:12:34,529 - root - INFO - settings/indexes params : storageMode=plasma 2021-02-14 21:12:34,534 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:34,534 - root - INFO - settings/web params on 172.23.104.246:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:34,584 - root - INFO - --> status:True 2021-02-14 21:12:34,584 - root - INFO - Done with init on 172.23.104.246. 2021-02-14 21:12:34,584 - root - INFO - running command.raw on 172.23.104.246: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:34,610 - root - INFO - command executed successfully with root 2021-02-14 21:12:34,610 - root - INFO - Done with cleanup on 172.23.104.246. 2021-02-14 21:12:35,813 - root - INFO - running command.raw on 172.23.104.249: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.249 -u Administrator -p password 2021-02-14 21:12:36,089 - root - INFO - command executed successfully with root 2021-02-14 21:12:37,100 - root - INFO - Setting KV memory quota as 2156 MB on 172.23.104.249 2021-02-14 21:12:37,100 - root - INFO - pools/default params : memoryQuota=2156 2021-02-14 21:12:37,109 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:37,109 - root - INFO - /node/controller/setupServices params on 172.23.104.249: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:37,114 - root - INFO - running command.raw on 172.23.104.237: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.237 -u Administrator -p password 2021-02-14 21:12:37,139 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:37,140 - root - INFO - settings/web params on 172.23.104.249:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:37,204 - root - INFO - --> status:True 2021-02-14 21:12:37,204 - root - INFO - Done with init on 172.23.104.249. 2021-02-14 21:12:37,205 - root - INFO - running command.raw on 172.23.104.249: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:37,230 - root - INFO - command executed successfully with root 2021-02-14 21:12:37,230 - root - INFO - Done with cleanup on 172.23.104.249. 2021-02-14 21:12:37,376 - root - INFO - command executed successfully with root 2021-02-14 21:12:38,388 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.104.237 2021-02-14 21:12:38,388 - root - INFO - pools/default params : memoryQuota=2147 2021-02-14 21:12:38,396 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:38,396 - root - INFO - /node/controller/setupServices params on 172.23.104.237: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:38,425 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:38,425 - root - INFO - settings/web params on 172.23.104.237:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:38,487 - root - INFO - --> status:True 2021-02-14 21:12:38,487 - root - INFO - Done with init on 172.23.104.237. 2021-02-14 21:12:38,487 - root - INFO - running command.raw on 172.23.104.237: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:38,506 - root - INFO - command executed successfully with root 2021-02-14 21:12:38,506 - root - INFO - Done with cleanup on 172.23.104.237. 2021-02-14 21:12:39,908 - root - INFO - running command.raw on 172.23.104.245: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.245 -u Administrator -p password 2021-02-14 21:12:40,150 - root - INFO - command executed successfully with root 2021-02-14 21:12:40,758 - root - INFO - running command.raw on 172.23.104.243: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.243 -u Administrator -p password 2021-02-14 21:12:40,945 - root - INFO - running command.raw on 172.23.104.247: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.247 -u Administrator -p password 2021-02-14 21:12:41,041 - root - INFO - command executed successfully with root 2021-02-14 21:12:41,161 - root - INFO - Setting KV memory quota as 12821 MB on 172.23.104.245 2021-02-14 21:12:41,161 - root - INFO - pools/default params : memoryQuota=12821 2021-02-14 21:12:41,168 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:41,168 - root - INFO - /node/controller/setupServices params on 172.23.104.245: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:41,196 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:41,196 - root - INFO - settings/web params on 172.23.104.245:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:41,231 - root - INFO - command executed successfully with root 2021-02-14 21:12:41,249 - root - INFO - --> status:True 2021-02-14 21:12:41,249 - root - INFO - Done with init on 172.23.104.245. 2021-02-14 21:12:41,249 - root - INFO - running command.raw on 172.23.104.245: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:41,265 - root - INFO - command executed successfully with root 2021-02-14 21:12:41,265 - root - INFO - Done with cleanup on 172.23.104.245. 2021-02-14 21:12:42,052 - root - INFO - Setting KV memory quota as 2156 MB on 172.23.104.243 2021-02-14 21:12:42,052 - root - INFO - pools/default params : memoryQuota=2156 2021-02-14 21:12:42,059 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:42,059 - root - INFO - /node/controller/setupServices params on 172.23.104.243: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:42,089 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:42,090 - root - INFO - settings/web params on 172.23.104.243:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:42,157 - root - INFO - --> status:True 2021-02-14 21:12:42,157 - root - INFO - Done with init on 172.23.104.243. 2021-02-14 21:12:42,157 - root - INFO - running command.raw on 172.23.104.243: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:42,176 - root - INFO - command executed successfully with root 2021-02-14 21:12:42,176 - root - INFO - Done with cleanup on 172.23.104.243. 2021-02-14 21:12:42,241 - root - INFO - Setting KV memory quota as 2156 MB on 172.23.104.247 2021-02-14 21:12:42,242 - root - INFO - pools/default params : memoryQuota=2156 2021-02-14 21:12:42,249 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:42,249 - root - INFO - /node/controller/setupServices params on 172.23.104.247: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:42,277 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:42,278 - root - INFO - settings/web params on 172.23.104.247:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:42,343 - root - INFO - --> status:True 2021-02-14 21:12:42,343 - root - INFO - Done with init on 172.23.104.247. 2021-02-14 21:12:42,344 - root - INFO - running command.raw on 172.23.104.247: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:42,359 - root - INFO - command executed successfully with root 2021-02-14 21:12:42,359 - root - INFO - Done with cleanup on 172.23.104.247. 2021-02-14 21:12:44,963 - root - INFO - running command.raw on 172.23.106.6: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.6 -u Administrator -p password 2021-02-14 21:12:45,238 - root - INFO - command executed successfully with root 2021-02-14 21:12:46,248 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.6 2021-02-14 21:12:46,248 - root - INFO - pools/default params : memoryQuota=2147 2021-02-14 21:12:46,255 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-02-14 21:12:46,255 - root - INFO - /node/controller/setupServices params on 172.23.106.6: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-02-14 21:12:46,285 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-02-14 21:12:46,285 - root - INFO - settings/web params on 172.23.106.6:8091:port=8091&username=Administrator&password=password 2021-02-14 21:12:46,356 - root - INFO - --> status:True 2021-02-14 21:12:46,356 - root - INFO - Done with init on 172.23.106.6. 2021-02-14 21:12:46,356 - root - INFO - running command.raw on 172.23.106.6: ls -td /tmp/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2021-02-14 21:12:46,374 - root - INFO - command executed successfully with root 2021-02-14 21:12:46,374 - root - INFO - Done with cleanup on 172.23.106.6. 2021-02-14 21:12:54,133 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-14 21:12:54,138 - root - INFO - node:172.23.104.246:8091 version:7.0.0-4454-enterprise aFamily:inet services:['index', 'kv', 'n1ql'] 2021-02-14 21:12:54,142 - root - INFO - node:172.23.104.248:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,148 - root - INFO - node:172.23.106.6:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,153 - root - INFO - node:172.23.104.237:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,159 - root - INFO - node:172.23.104.247:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,163 - root - INFO - node:172.23.104.249:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,168 - root - INFO - node:172.23.104.243:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,173 - root - INFO - node:172.23.104.245:8091 version:7.0.0-4454-enterprise aFamily:inet services:['kv'] 2021-02-14 21:12:54,173 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-14 21:12:54,173 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-14 21:12:54,173 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.246 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.248 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.106.6 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.237 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.247 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.249 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.243 2021-02-14 21:12:54,173 - root - INFO - INSTALL COMPLETED ON: 172.23.104.245 2021-02-14 21:12:54,173 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-14 21:12:54,174 - root - INFO - TOTAL INSTALL TIME = 327 seconds success INFO:root:SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.104.248 as root INFO:root:SSH Connected to 172.23.104.237 as root INFO:root:SSH Connected to 172.23.106.6 as root INFO:root:SSH Connected to 172.23.104.249 as root INFO:root:SSH Connected to 172.23.104.247 as root INFO:root:SSH Connected to 172.23.104.245 as root INFO:root:SSH Connected to 172.23.104.243 as root INFO:root:SSH Connected to 172.23.104.246 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.248: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.246: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.237: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.249: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.245: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.247: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.6: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.243: iptables -F INFO:root:command executed successfully with root 172.23.104.246 172.23.104.248 172.23.104.249 172.23.104.237 172.23.104.245 172.23.104.247 172.23.106.6 172.23.104.243 Need to set ALLOW_HTP back to True to do git pull branch Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out '802240fa3c0aeca3ba88203661de10b072de131c' Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.sonic.net * extras: repos.lax.quadranet.com * updates: mirrors.sonic.net Package maven-3.0.5-17.el7.noarch already installed and latest version Nothing to do find: ‘/root/workspace/*/logs/*’: No such file or directory find: ‘/root/workspace/’: No such file or directory python3: no process found python3 testrunner.py -i /tmp/testexec.110140.ini -c eventing/eventing_rebalance.conf -p get-cbcollect-info=True,GROUP=bucket_op_with_timers -d failed=http://qa.sc.couchbase.com/job/test_suite_executor/314534/ INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: eventing/eventing_rebalance.conf INFO:__main__:Test prefix: eventing.eventing_rebalance.EventingRebalance INFO:__main__:Downloading http://qa.sc.couchbase.com/job/test_suite_executor/314534//testReport/api/xml?pretty=true to logs/test_suite_executor_314534__testresult.xml INFO:__main__:Loading result data from logs/test_suite_executor_314534__testresult.xml INFO:__main__:-- logs/test_suite_executor_314534__testresult.xml -- INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 1 INFO:__main__:--> Running test: eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers,get-cbcollect-info=True,skip_log_scan=False,bucket_storage=couchstore INFO:__main__:Logs folder: /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers/logs/testrunner-21-Feb-14_21-13-01/test_1 *** TestRunner *** {'GROUP': 'bucket_op_with_timers', 'cluster_name': 'testexec.110140', 'conf_file': 'eventing/eventing_rebalance.conf', 'get-cbcollect-info': 'True', 'ini': '/tmp/testexec.110140.ini', 'num_nodes': 8, 'spec': 'eventing_rebalance'} Only cases in GROUPs 'bucket_op_with_timers' will be executed Logs will be stored at /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers/logs/testrunner-21-Feb-14_21-13-01/test_1 ./testrunner -i /tmp/testexec.110140.ini -p get-cbcollect-info=True,GROUP=bucket_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers,get-cbcollect-info=True,skip_log_scan=False,bucket_storage=couchstore Test Input params: {'doc-per-day': '5', 'dataset': 'default', 'nodes_init': '5', 'services_init': 'kv-kv-kv-eventing-index:n1ql', 'groups': 'simple', 'reset_services': 'True,', 'handler_code': 'bucket_op_with_timers', 'GROUP': 'bucket_op_with_timers', 'get-cbcollect-info': 'True', 'skip_log_scan': 'False', 'bucket_storage': 'couchstore', 'ini': '/tmp/testexec.110140.ini', 'cluster_name': 'testexec.110140', 'spec': 'eventing_rebalance', 'conf_file': 'eventing/eventing_rebalance.conf', 'num_nodes': 8, 'case_number': 1, 'logs_folder': '/data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers/logs/testrunner-21-Feb-14_21-13-01/test_1'} Run before suite setup for eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes suite_setUp (eventing.eventing_rebalance.EventingRebalance) ... -->before_suite_name:eventing.eventing_rebalance.EventingRebalance.suite_setUp,suite: ]> 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [eventing_base.setUp] Starting Test: suite_setUp 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4454-enterprise 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster ['7.0.0-4454-enterprise'] 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.246, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 1.757469244288225, 'mem_free': 3207380992, 'mem_total': 3705745408, 'swap_mem_used': 139370496, 'swap_mem_total': 3909087232} 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:13:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:13:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:13:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:13:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:13:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:13:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:13:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:13:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:13:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:13:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.246, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:13:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:13:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:13 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:13 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:13:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:13:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:15 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:13:15 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:20 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:13:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:13:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:25 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 25796 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:13:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 25985 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:13:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:27 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:27 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:13:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:13:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:28 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:13:28 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:13:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:13:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:13:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:13:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:34 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 16589 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 16778 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:13:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:13:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:13:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:13:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:41 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:41 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:13:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:13:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:42 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:13:42 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:13:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:13:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:13:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:48 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 98468 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 98610 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:13:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:13:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:13:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:13:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:13:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:13:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:13:55 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:13:55 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:13:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:13:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:13:57 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:13:57 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:02 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:14:03 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:14:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:14:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:03 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 82293 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 82437 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:14:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:14:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:09 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:09 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:14:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:11 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:14:11 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:14:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:17 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26298 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 26442 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:14:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:14:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:23 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:23 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:14:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:14:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:25 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:14:25 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:30 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:31 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:14:31 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:14:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:14:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:31 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 28168 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 28313 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:14:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:14:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:37 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:37 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:14:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:14:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:39 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:14:39 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:14:44 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:45 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 23961 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 24105 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:14:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:14:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:14:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:53 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:14:53 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:14:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:14:59 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 95358 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 95508 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:15:04 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:15:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:15:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:15 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:15 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:15:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:15:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:17 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:15:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:22 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 26506 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 26695 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:15:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:15:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:15:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:15:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:15:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:29 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:29 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:15:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:15:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:30 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:15:30 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:15:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:15:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:15:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:15:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:36 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 17302 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 17491 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:15:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:15:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:15:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:15:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:43 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:43 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:15:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:15:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:44 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:15:44 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:15:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:15:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:15:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:15:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:50 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 99217 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 99361 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:15:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:15:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:15:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:15:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:15:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:15:57 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:15:57 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:15:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:15:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:15:58 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:15:58 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:16:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:16:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:03 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:04 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 83052 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 83196 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:16:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:16:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:16:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:16:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:16:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:11 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:11 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:16:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:16:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:13 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:16:13 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:18 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:16:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:16:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:23 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 27072 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:16:23 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 27215 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:16:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:16:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:25 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:25 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:16:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:16:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:27 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:16:27 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:32 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:16:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:16:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:37 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 28930 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:16:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 29074 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:16:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:16:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:39 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:39 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:16:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:16:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:41 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:16:41 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:47 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24722 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 24866 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:16:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:16:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:16:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:16:53 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:16:53 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:16:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:16:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:16:55 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:16:55 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:01 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 96175 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 96326 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:17:06 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.246 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '109', 'memoryTotal': 3705745408, 'memoryFree': 3260465152, 'mcdMemoryReserved': 2827, 'mcdMemoryAllocated': 2827, 'status': 'healthy', 'hostname': '172.23.104.246:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1454, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.246', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3534, 'curr_items': 0} 2021-02-14 21:17:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.104.246,8091,['kv']) 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.104.246: 8091:hostname=172.23.104.246&user=Administrator&password=password&services=kv 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.246:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:17:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.246:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.246:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.248 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '94', 'memoryTotal': 3705745408, 'memoryFree': 3182530560, 'mcdMemoryReserved': 2827, 'mcdMemoryAllocated': 2827, 'status': 'healthy', 'hostname': '172.23.104.248:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1454, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.248', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3534, 'curr_items': 0} 2021-02-14 21:17:17 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.248:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.248:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.248:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.248:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.6 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '84', 'memoryTotal': 4201840640, 'memoryFree': 3627909120, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.106.6:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.106.6', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-14 21:17:17 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.6:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.6:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.6:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.6:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.237 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '70', 'memoryTotal': 4201848832, 'memoryFree': 3644256256, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.104.237:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.237', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-14 21:17:18 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.237:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.237:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:17:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.247 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '55', 'memoryTotal': 4219441152, 'memoryFree': 3658280960, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.247:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.247', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:17:19 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.247:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.247:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:17:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.249 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '44', 'memoryTotal': 4219445248, 'memoryFree': 3679531008, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.249:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.249', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:17:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.249:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.249:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:17:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.243 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '30', 'memoryTotal': 4219445248, 'memoryFree': 3709812736, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.243:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.243', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:17:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.243:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.243:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:17:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.243:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.243:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.245 port:8091 ssh_username:root, nodes/self 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '19', 'memoryTotal': 25084071936, 'memoryFree': 24288010240, 'mcdMemoryReserved': 19137, 'mcdMemoryAllocated': 19137, 'status': 'healthy', 'hostname': '172.23.104.245:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 13687, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.245', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 23922, 'curr_items': 0} 2021-02-14 21:17:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.245:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=12821 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.245:8091:port=8091&username=Administrator&password=password 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.245:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.245:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:17:22 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:17:22 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.246 **** 2021-02-14 21:17:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.104.246:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-02-14 21:17:22 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-02-14 21:17:23 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:17:28 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-02-14 21:17:28 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-02-14 21:17:28 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.248:8091 to cluster 2021-02-14 21:17:28 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.248:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:17:31 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.6:8091 to cluster 2021-02-14 21:17:31 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.6:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:17:34 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.237:8091 to cluster 2021-02-14 21:17:34 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.237:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:17:37 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.247:8091 to cluster 2021-02-14 21:17:37 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.247:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:17:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:17:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:17:40 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:17:40 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-02-14 21:17:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-02-14 21:17:50 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.030500650405884 sec 2021-02-14 21:17:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:17:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.246 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:17:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:17:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:17:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:17:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:17:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:17:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:17:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:17:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:55 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:17:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:17:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:17:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:17:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.237:8091 => {'services': ['eventing'], 'cpu_utilization': 1.404917210235825, 'mem_free': 3610914816, 'mem_total': 4201848832, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['kv'], 'cpu_utilization': 5.614300100704934, 'mem_free': 3214815232, 'mem_total': 3705745408, 'swap_mem_used': 138973184, 'swap_mem_total': 3909087232} 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.247:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.8038181361466968, 'mem_free': 3646840832, 'mem_total': 4219441152, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.248:8091 => {'services': ['kv'], 'cpu_utilization': 3.690685413005272, 'mem_free': 3156402176, 'mem_total': 3705745408, 'swap_mem_used': 0, 'swap_mem_total': 3909087232} 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.6:8091 => {'services': ['kv'], 'cpu_utilization': 2.71424981151043, 'mem_free': 3597623296, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:18:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:18:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2021-02-14 21:18:03 | INFO | MainProcess | MainThread | [eventing_rebalance.setUp] 100 2021-02-14 21:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:18:03 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2021-02-14 21:18:03 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:05 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:05 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:18:05 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:06 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:06 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:07 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:08 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:18:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:18:08 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:18:09 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:18:09 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2021-02-14 21:18:09 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:18:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:12 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:18:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:18:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:18:15 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:18:15 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2021-02-14 21:18:15 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:18:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:16 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:17 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:18 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:18 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:18:18 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:19 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:20 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:18:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:18:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:18:21 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:18:21 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:18:21 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:18:21 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:18:21 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 485.637689ms 2021-02-14 21:18:21 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI 2021-02-14 21:18:21 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.822226023s 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 25.269496ms 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.533727ms 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI 2021-02-14 21:18:23 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.904588177s 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 30.33229ms 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.209752ms 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI 2021-02-14 21:18:25 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.75043355s 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.880496ms 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.104.237 is [] 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:18:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+metadata 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [eventing_base.tearDown] number of documents in metadata bucket 0 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.237:8091 => {'services': ['eventing'], 'cpu_utilization': 2.261874842925358, 'mem_free': 3571060736, 'mem_total': 4201848832, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['kv'], 'cpu_utilization': 16.63662116919907, 'mem_free': 2893520896, 'mem_total': 3705745408, 'swap_mem_used': 138964992, 'swap_mem_total': 3909087232} 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.247:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 8.409321175278622, 'mem_free': 3405811712, 'mem_total': 4219441152, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.248:8091 => {'services': ['kv'], 'cpu_utilization': 14.57583547557841, 'mem_free': 2839642112, 'mem_total': 3705745408, 'swap_mem_used': 0, 'swap_mem_total': 3909087232} 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.6:8091 => {'services': ['kv'], 'cpu_utilization': 12.88515406162465, 'mem_free': 3463307264, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:18:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:18:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:18:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:18:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:18:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:18:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:18:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:18:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:18:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:18:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:18:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:18:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2021-02-14 21:18:38 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.104.246 2021-02-14 21:18:39 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:18:39 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : ['metadata', 'src_bucket'] 2021-02-14 21:18:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 172.23.104.246 2021-02-14 21:18:40 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:18:40 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : ['src_bucket'] 2021-02-14 21:18:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 172.23.104.246 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : [] 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 172.23.104.246 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'ejectedNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:18:41 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:18:51 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-02-14 21:19:01 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-02-14 21:19:21 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2021-02-14 21:19:21 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.104.246 port:8091 ssh_username:root ? [('ns_1@172.23.104.237', 8091), ('ns_1@172.23.104.247', 8091), ('ns_1@172.23.104.248', 8091), ('ns_1@172.23.106.6', 8091)] 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:19:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:19:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:19:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:34 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:19:34 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:19:39 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:19:40 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:19:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:19:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:40 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 30685 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 30874 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:19:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:19:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:46 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:19:46 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:19:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:19:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:47 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:19:47 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:19:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:19:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:52 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:53 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 21235 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 21424 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:19:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:19:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:19:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:19:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:19:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:19:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:00 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:00 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:20:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:20:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:02 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:20:02 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:07 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:20:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:20:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:20:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:08 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 103216 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 103358 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:20:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:20:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:14 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:14 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:20:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:20:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:16 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:20:16 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:20:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:22 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 87306 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 87450 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:20:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:20:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:20:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:30 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:20:30 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:20:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:36 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 31292 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 31434 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:20:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:20:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:20:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:44 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:20:44 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:20:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:20:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:20:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:50 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 32444 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 328 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:20:55 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:20:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:20:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:20:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:20:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:20:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:20:57 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:20:57 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:20:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:20:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:20:59 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:20:59 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:21:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:05 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 28143 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 28286 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:21:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:21:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:21:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:13 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:21:13 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:21:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:19 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 99745 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 99895 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:21:24 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... ok ---------------------------------------------------------------------- Ran 1 test in 512.374s OK test_eventing_rebalance_with_multiple_kv_nodes (eventing.eventing_rebalance.EventingRebalance) ... Cluster instance shutdown with force -->result: 2021-02-14 21:21:34 | INFO | MainProcess | test_thread | [eventing_base.setUp] Starting Test: test_eventing_rebalance_with_multiple_kv_nodes 2021-02-14 21:21:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:21:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:21:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4454-enterprise 2021-02-14 21:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_eventing_rebalance_with_multiple_kv_nodes============== 2021-02-14 21:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.246, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-02-14 21:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-02-14 21:21:35 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:21:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:21:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:21:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:21:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:21:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:21:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-02-14 21:21:45 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.246, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:21:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:46 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:21:47 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:21:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:21:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:48 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:21:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:21:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:54 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 735 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 938 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:21:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:22:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:22:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:02 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:22:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:07 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:22:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 23307 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 23496 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:22:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:22:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:14 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:14 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:22:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:22:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:16 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:22:16 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:21 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:22:22 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:22:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:22:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:22 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 105223 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 105365 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:22:27 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:22:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:28 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:28 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:22:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:22:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:30 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:22:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:35 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 89416 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 89560 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:22:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:22:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:42 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:42 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:22:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:22:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:44 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:22:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:22:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:22:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:50 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 979 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 1122 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:22:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:22:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:22:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:22:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:22:56 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:22:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:22:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:22:58 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:22:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:03 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:23:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:23:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 2092 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:23:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 2240 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:23:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:23:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:10 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:10 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:23:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:23:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:12 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:23:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:17 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:23:18 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:23:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:23:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:18 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 30148 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 30294 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:23:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:23:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:24 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:24 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:23:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:23:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:26 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:23:26 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:31 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:32 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:23:32 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:23:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:23:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:32 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 101864 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 102014 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:23:37 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:23:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:23:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:48 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:23:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:23:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:49 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:23:49 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:23:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:23:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:23:55 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 1475 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 1667 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:24:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:24:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:24:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:24:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:02 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:02 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:24:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:24:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:03 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:24:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:24:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:09 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 24020 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 24209 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:24:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:24:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:24:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:17 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:24:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:24:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:24:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:24:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:24:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:23 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 105977 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 106119 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:24:28 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:24:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:24:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:24:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:30 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:30 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:24:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:24:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:31 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:24:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:24:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:24:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:24:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:24:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:37 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 90177 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 90321 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:24:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:24:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:24:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:24:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:44 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:44 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:24:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:24:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:45 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:24:45 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:24:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:24:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:24:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:51 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 1732 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 1874 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:24:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:24:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:24:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:24:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:24:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:24:58 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:24:58 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:24:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:24:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:24:59 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:24:59 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:25:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:05 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:25:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:25:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:10 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 2857 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:25:10 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 3001 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:25:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:25:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:12 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:25:12 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:25:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:25:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:14 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:25:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:25:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:20 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 30911 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 31055 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:25:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:25:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:26 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:25:26 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:25:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:25:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:28 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:25:28 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:34 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 102683 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 102833 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:25:39 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.246 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '109', 'memoryTotal': 3705745408, 'memoryFree': 3248992256, 'mcdMemoryReserved': 2827, 'mcdMemoryAllocated': 2827, 'status': 'healthy', 'hostname': '172.23.104.246:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1454, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.246', 'ip': '172.23.104.246', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3534, 'curr_items': 0} 2021-02-14 21:25:49 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.104.246,8091,['kv']) 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.104.246: 8091:hostname=172.23.104.246&user=Administrator&password=password&services=kv 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.246:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:25:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.246:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.246:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.248 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '94', 'memoryTotal': 3705745408, 'memoryFree': 3176869888, 'mcdMemoryReserved': 2827, 'mcdMemoryAllocated': 2827, 'status': 'healthy', 'hostname': '172.23.104.248:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1454, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.248', 'ip': '172.23.104.248', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3534, 'curr_items': 0} 2021-02-14 21:25:50 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.248:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.248:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.248:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.248:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.6 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '85', 'memoryTotal': 4201840640, 'memoryFree': 3627593728, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.106.6:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.106.6', 'ip': '172.23.106.6', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-14 21:25:50 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.6:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.6:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:25:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.6:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.6:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.237 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '70', 'memoryTotal': 4201848832, 'memoryFree': 3658440704, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.104.237:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.237', 'ip': '172.23.104.237', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-14 21:25:51 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.237:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.237:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:25:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.247 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '55', 'memoryTotal': 4219441152, 'memoryFree': 3671781376, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.247:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.247', 'ip': '172.23.104.247', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:25:52 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.247:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.247:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:25:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.249 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '45', 'memoryTotal': 4219445248, 'memoryFree': 3680538624, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.249:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.249', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:25:53 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.249:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.249:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:25:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.243 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '29', 'memoryTotal': 4219445248, 'memoryFree': 3707531264, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.243:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.243', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:25:54 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.243:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.243:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.243:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.243:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.245 port:8091 ssh_username:root, nodes/self 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '14', 'memoryTotal': 25084071936, 'memoryFree': 24293203968, 'mcdMemoryReserved': 19137, 'mcdMemoryAllocated': 19137, 'status': 'healthy', 'hostname': '172.23.104.245:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 13687, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.245', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 23922, 'curr_items': 0} 2021-02-14 21:25:55 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.245:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=12821 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.245:8091:port=8091&username=Administrator&password=password 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.245:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.245:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:25:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:25:55 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.246 **** 2021-02-14 21:25:55 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.104.246:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-02-14 21:25:55 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-02-14 21:25:55 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:26:00 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-02-14 21:26:00 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2021-02-14 21:26:01 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.248:8091 to cluster 2021-02-14 21:26:01 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.248:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:26:03 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.6:8091 to cluster 2021-02-14 21:26:03 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.6:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:26:06 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.237:8091 to cluster 2021-02-14 21:26:06 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.237:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:26:08 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.247:8091 to cluster 2021-02-14 21:26:08 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.247:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:26:11 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:26:11 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:26:11 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:26:11 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-02-14 21:26:21 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-02-14 21:26:21 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.026477575302124 sec 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.246 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-02-14 21:26:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:26:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:26:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:26:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:26:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:26:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:26:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:26:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:26:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:26:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:27 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.237:8091 => {'services': ['eventing'], 'cpu_utilization': 1.025, 'mem_free': 3647520768, 'mem_total': 4201848832, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['kv'], 'cpu_utilization': 6.268806419257773, 'mem_free': 3200643072, 'mem_total': 3705745408, 'swap_mem_used': 138498048, 'swap_mem_total': 3909087232} 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.247:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.228994231251568, 'mem_free': 3659128832, 'mem_total': 4219441152, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.248:8091 => {'services': ['kv'], 'cpu_utilization': 2.914572864321608, 'mem_free': 3153653760, 'mem_total': 3705745408, 'swap_mem_used': 0, 'swap_mem_total': 3909087232} 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.6:8091 => {'services': ['kv'], 'cpu_utilization': 6.710228700678562, 'mem_free': 3601944576, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:26:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2021-02-14 21:26:34 | INFO | MainProcess | test_thread | [eventing_rebalance.setUp] 100 2021-02-14 21:26:35 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:26:35 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2021-02-14 21:26:35 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:36 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:37 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:38 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:38 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:26:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:39 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:26:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:26:40 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:26:40 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:26:41 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:26:41 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2021-02-14 21:26:41 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:42 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:43 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:44 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:45 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:45 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:26:45 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:26:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:26:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:26:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:26:46 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:26:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:26:47 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2021-02-14 21:26:47 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:26:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:49 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:49 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:26:49 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:50 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:51 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:52 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:26:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:26:52 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:26:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:26:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:26:53 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:26:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 483.918018ms 2021-02-14 21:26:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI 2021-02-14 21:26:53 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.84137921s 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.227585ms 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.560167ms 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI 2021-02-14 21:26:55 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.876428531s 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.500945ms 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.223817ms 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI 2021-02-14 21:26:57 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI 2021-02-14 21:26:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.821340658s 2021-02-14 21:26:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:26:59 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:26:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 23.789271ms 2021-02-14 21:27:00 | INFO | MainProcess | test_thread | [eventing_base.create_save_function_body] saving function b'{\n "code": 0,\n "info": {\n "status": "Stored function: \'Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes\' in metakv",\n "warnings": null\n }\n}' 2021-02-14 21:27:00 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] Deploying the following handler code : Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'access': 'rw'}], 'metadata_bucket': 'metadata', 'source_bucket': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.5.0'} 2021-02-14 21:27:00 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { delete dst_bucket[context.docID]; } function timerCallback(context) { dst_bucket[context.docID] = context.random_text; } 2021-02-14 21:27:02 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] deploy Application : b'' 2021-02-14 21:27:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... 2021-02-14 21:27:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... 2021-02-14 21:27:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... 2021-02-14 21:27:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. Waiting for Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... 2021-02-14 21:27:22 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:27:22 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:27:22 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:27:23 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.249:8091 to cluster 2021-02-14 21:27:23 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.249:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:27:25 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.0063555240631104 seconds 2021-02-14 21:27:25 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.243:8091 to cluster 2021-02-14 21:27:25 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.243:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:27:29 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.243,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.104.249,ns_1@172.23.106.6', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:27:29 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:27:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:27:29 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-02-14 21:27:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 38.00 % 2021-02-14 21:27:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 38.00% 2021-02-14 21:27:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 78.00 % 2021-02-14 21:27:49 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 78.00% 2021-02-14 21:27:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 78.00 % 2021-02-14 21:27:59 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 78.00% 2021-02-14 21:28:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 83.00 % 2021-02-14 21:28:09 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 83.00% 2021-02-14 21:28:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 88.00 % 2021-02-14 21:28:19 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 88.00% 2021-02-14 21:28:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 90.00 % 2021-02-14 21:28:29 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 90.00% 2021-02-14 21:28:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-02-14 21:28:39 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 70.20985531806946 sec 2021-02-14 21:28:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:28:39 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:28:40 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9856 Expected : 10080 ... 2021-02-14 21:29:12 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Documents in source bucket : 10080 2021-02-14 21:29:12 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Final docs count... Current : 10080 Expected : 10080 2021-02-14 21:29:13 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.104.237 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 14982, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 10080, "dcp_mutation_sent_to_worker": 10080, "dcp_snapshot": 10079, "dcp_stream_req_counter": 2293, "dcp_stream_req_err_counter": 10, "dcp_streamend": 1030, "dcp_streamreq": 2283, "execution_stats": 360, "failure_stats": 360, "latency_stats": 360, "lcb_exception_stats": 360, "log_level": 3, "num_processed_events": 10080, "processed_events_size": 7254832, "thr_count": 3, "thr_map": 3, "timer_events": 10080, "timer_responses_received": 10080, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 10080, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 10080, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 14970, "num_processed_events": 10080, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 10080, "processed_events_size": 7254832, "timer_callback_failure": 0, "timer_callback_success": 10080, "timer_cancel_counter": 0, "timer_create_counter": 10080, "timer_create_failure": 0, "timer_msg_counter": 10080, "timer_responses_sent": 0, "timestamp": { "92566": "2021-02-15T05:29:12Z", "92575": "2021-02-15T05:29:12Z", "92584": "2021-02-15T05:29:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "92566": "2021-02-15T05:29:12Z", "92575": "2021-02-15T05:29:12Z", "92584": "2021-02-15T05:29:12Z" }, "v8worker_events_lost": 0 }, "function_id": 315539844, "function_name": "Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 32, "internal_vb_distribution_stats": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 17000, "50": 1300, "80": 1500, "90": 1700, "95": 2100, "99": 5100 }, "lcb_creds_request_counter": 66, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.104.237:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.104.237:8096": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": 92566, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": 92575, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": 92584 } } ] 2021-02-14 21:29:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:29:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.243:11210 src_bucket 2021-02-14 21:29:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:29:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.249:11210 src_bucket 2021-02-14 21:29:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:29:15 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.243,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.104.249,ns_1@172.23.106.6', 'ejectedNodes': 'ns_1@172.23.104.249,ns_1@172.23.104.243', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:29:15 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:29:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:29:15 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-02-14 21:29:17 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.0054690837860107 seconds 2021-02-14 21:29:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 23.00 % 2021-02-14 21:29:25 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 23.00% 2021-02-14 21:29:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 47.00 % 2021-02-14 21:29:35 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 47.00% 2021-02-14 21:29:45 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 55.00 % 2021-02-14 21:29:45 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 55.00% 2021-02-14 21:29:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 71.00 % 2021-02-14 21:29:55 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 71.00% 2021-02-14 21:30:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 78.00 % 2021-02-14 21:30:05 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 78.00% 2021-02-14 21:30:15 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-02-14 21:30:15 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://172.23.104.249:8091/nodes/self error [Errno 111] Connection refused 2021-02-14 21:30:18 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 63.19266128540039 sec 2021-02-14 21:30:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:30:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:30:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 2864 Expected : 0 ... 2021-02-14 21:30:51 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Documents in source bucket : 0 2021-02-14 21:30:51 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Final docs count... Current : 0 Expected : 0 2021-02-14 21:30:53 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.104.237 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 27370, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 10080, "dcp_deletion_sent_to_worker": 10080, "dcp_mutation": 10080, "dcp_mutation_sent_to_worker": 10080, "dcp_snapshot": 20154, "dcp_stream_req_counter": 29035, "dcp_stream_req_err_counter": 26240, "dcp_streamend": 1440, "dcp_streamreq": 2795, "execution_stats": 657, "failure_stats": 657, "latency_stats": 657, "lcb_exception_stats": 657, "log_level": 3, "num_processed_events": 20160, "processed_events_size": 10559340, "thr_count": 3, "thr_map": 3, "timer_events": 20160, "timer_responses_received": 20160, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 10080, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 10080, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 10080, "enqueued_dcp_mutation_msg_counter": 10080, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 27358, "num_processed_events": 20160, "on_delete_failure": 0, "on_delete_success": 10080, "on_update_failure": 0, "on_update_success": 10080, "processed_events_size": 10559340, "timer_callback_failure": 0, "timer_callback_success": 20160, "timer_cancel_counter": 0, "timer_create_counter": 20160, "timer_create_failure": 0, "timer_msg_counter": 20160, "timer_responses_sent": 0, "timestamp": { "92566": "2021-02-15T05:30:51Z", "92575": "2021-02-15T05:30:51Z", "92584": "2021-02-15T05:30:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "92566": "2021-02-15T05:30:51Z", "92575": "2021-02-15T05:30:51Z", "92584": "2021-02-15T05:30:51Z" }, "v8worker_events_lost": 0 }, "function_id": 315539844, "function_name": "Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 52, "internal_vb_distribution_stats": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 20600, "50": 1300, "80": 1700, "90": 2200, "95": 3400, "99": 7800 }, "lcb_creds_request_counter": 68, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.104.237:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.104.237:8096": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": 92566, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": 92575, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": 92584 } } ] 2021-02-14 21:30:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.246 port:8091 ssh_username:root, ip:172.23.104.248 port:8091 ssh_username:root, ip:172.23.106.6 port:8091 ssh_username:root] 2021-02-14 21:30:53 | INFO | MainProcess | test_thread | [rest_client.add_node] adding remote node @172.23.104.249:8091 to this cluster @172.23.104.237:8091 2021-02-14 21:30:55 | INFO | MainProcess | test_thread | [rest_client.add_node] adding remote node @172.23.104.243:8091 to this cluster @172.23.104.237:8091 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.243 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.249 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.243 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.249 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.243 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.249 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.243 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.249 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.243 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.249 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.243 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 172.23.104.249 not part of cluster inactiveAdded 2021-02-14 21:30:58 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.243,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.104.249,ns_1@172.23.106.6', 'ejectedNodes': 'ns_1@172.23.104.248,ns_1@172.23.106.6', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:30:58 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:30:58 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:30:58 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-02-14 21:31:01 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.0057215690612793 seconds 2021-02-14 21:31:09 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 19.00 % 2021-02-14 21:31:09 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 19.00% 2021-02-14 21:31:19 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 23.00 % 2021-02-14 21:31:19 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 23.00% 2021-02-14 21:31:29 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 42.00 % 2021-02-14 21:31:29 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 42.00% 2021-02-14 21:31:39 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 53.00 % 2021-02-14 21:31:39 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 53.00% 2021-02-14 21:31:49 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 66.00 % 2021-02-14 21:31:49 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 66.00% 2021-02-14 21:31:59 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 78.00 % 2021-02-14 21:31:59 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 78.00% 2021-02-14 21:32:09 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-02-14 21:32:09 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://172.23.104.248:8091/nodes/self error [Errno 111] Connection refused 2021-02-14 21:32:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://172.23.104.248:8091/nodes/self error [Errno 111] Connection refused 2021-02-14 21:32:18 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 79.38610339164734 sec 2021-02-14 21:32:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:32:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:32:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 8836 Expected : 10080 ... 2021-02-14 21:32:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9736 Expected : 10080 ... 2021-02-14 21:33:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:33:53 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:34:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:34:56 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:35:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:35:58 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:36:30 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:37:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:37:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:38:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:38:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:39:06 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : dst_bucket 2021-02-14 21:39:06 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2021-02-14 21:39:06 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : metadata 2021-02-14 21:39:06 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2021-02-14 21:39:06 | INFO | MainProcess | test_thread | [eventing_base.bucket_compaction] Compacting bucket : src_bucket 2021-02-14 21:39:06 | INFO | MainProcess | test_thread | [rest_client.compact_bucket] bucket compaction successful 2021-02-14 21:39:07 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:39:38 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:40:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:40:41 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:41:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:41:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:42:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:42:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:43:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... 2021-02-14 21:43:50 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Documents in source bucket : 10080 2021-02-14 21:43:51 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.104.237 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 11, "agg_messages_sent_to_worker": 50502, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 10080, "dcp_deletion_sent_to_worker": 10080, "dcp_mutation": 20160, "dcp_mutation_sent_to_worker": 20160, "dcp_snapshot": 30234, "dcp_stream_req_counter": 56170, "dcp_stream_req_err_counter": 52266, "dcp_streamend": 2122, "dcp_streamreq": 3904, "execution_stats": 2994, "failure_stats": 2994, "latency_stats": 2994, "lcb_exception_stats": 2994, "log_level": 3, "num_processed_events": 30240, "processed_events_size": 17822664, "thr_count": 3, "thr_map": 3, "timer_events": 30240, "timer_responses_received": 30240, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 10080, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20160, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 10080, "enqueued_dcp_mutation_msg_counter": 20160, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 50490, "num_processed_events": 30240, "on_delete_failure": 0, "on_delete_success": 10080, "on_update_failure": 0, "on_update_success": 20160, "processed_events_size": 17822664, "timer_callback_failure": 1, "timer_callback_success": 30239, "timer_cancel_counter": 0, "timer_create_counter": 30240, "timer_create_failure": 0, "timer_msg_counter": 30240, "timer_responses_sent": 0, "timestamp": { "92566": "2021-02-15T05:43:50Z", "92575": "2021-02-15T05:43:50Z", "92584": "2021-02-15T05:43:50Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "92566": "2021-02-15T05:43:50Z", "92575": "2021-02-15T05:43:50Z", "92584": "2021-02-15T05:43:50Z" }, "v8worker_events_lost": 0 }, "function_id": 315539844, "function_name": "Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 23100, "50": 1400, "80": 1900, "90": 2600, "95": 4200, "99": 9500 }, "lcb_creds_request_counter": 102, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.104.237:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.104.237:8096": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": 92566, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": 92575, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": 92584 } } ] 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select RAW count(*) from dst_bucket 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW+count%28%2A%29+from+dst_bucket 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.110365ms 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_document_count_via_index] ================== number of doc in dst_bucket via index =========================== 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_document_count_via_index] [10079] 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_document_count_via_index] =============================================================================================== 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select RAW count(0) from metadata where meta().id like 'eventing:%:al%' 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW+count%280%29+from+metadata+where+meta%28%29.id+like+%27eventing%3A%25%3Aal%25%27 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 27.519352ms 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_timer_alarm_context] ================== alarm documents in metadata ================================================ 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_timer_alarm_context] {'requestID': 'fe1b50df-01c1-4125-af7e-28f80709202f', 'signature': 'number', 'results': [0], 'status': 'success', 'metrics': {'elapsedTime': '27.519352ms', 'executionTime': '27.41101ms', 'resultCount': 1, 'resultSize': 1, 'serviceLoad': 6}} 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_timer_alarm_context] =============================================================================================== 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY select RAW count(0) from metadata where meta().id like 'eventing:%:cx%' 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=select+RAW+count%280%29+from+metadata+where+meta%28%29.id+like+%27eventing%3A%25%3Acx%25%27 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.041138ms 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_timer_alarm_context] ================== context documents in metadata ============================================= 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_timer_alarm_context] {'requestID': '9bbe55b6-325d-4ba3-852a-c96b1799aedc', 'signature': 'number', 'results': [0], 'status': 'success', 'metrics': {'elapsedTime': '21.041138ms', 'executionTime': '20.949098ms', 'resultCount': 1, 'resultSize': 1, 'serviceLoad': 6}} 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_timer_alarm_context] =============================================================================================== 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.104.237 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 11, "agg_messages_sent_to_worker": 50532, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 10080, "dcp_deletion_sent_to_worker": 10080, "dcp_mutation": 20160, "dcp_mutation_sent_to_worker": 20160, "dcp_snapshot": 30234, "dcp_stream_req_counter": 56170, "dcp_stream_req_err_counter": 52266, "dcp_streamend": 2122, "dcp_streamreq": 3904, "execution_stats": 3000, "failure_stats": 3000, "latency_stats": 3000, "lcb_exception_stats": 3000, "log_level": 3, "num_processed_events": 30240, "processed_events_size": 17822664, "thr_count": 3, "thr_map": 3, "timer_events": 30240, "timer_responses_received": 30240, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 10080, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20160, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 10080, "enqueued_dcp_mutation_msg_counter": 20160, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 50520, "num_processed_events": 30240, "on_delete_failure": 0, "on_delete_success": 10080, "on_update_failure": 0, "on_update_success": 20160, "processed_events_size": 17822664, "timer_callback_failure": 1, "timer_callback_success": 30239, "timer_cancel_counter": 0, "timer_create_counter": 30240, "timer_create_failure": 0, "timer_msg_counter": 30240, "timer_responses_sent": 0, "timestamp": { "92566": "2021-02-15T05:43:52Z", "92575": "2021-02-15T05:43:52Z", "92584": "2021-02-15T05:43:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "92566": "2021-02-15T05:43:52Z", "92575": "2021-02-15T05:43:52Z", "92584": "2021-02-15T05:43:52Z" }, "v8worker_events_lost": 0 }, "function_id": 315539844, "function_name": "Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 23100, "50": 1400, "80": 1900, "90": 2600, "95": 4200, "99": 9500 }, "lcb_creds_request_counter": 102, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.104.237:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.104.237:8096": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_0": 92566, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_1": 92575, "worker_Function_551935272_test_eventing_rebalance_with_multiple_kv_nodes_2": 92584 } } ] 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-02-14 21:43:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.237:8091 => {'services': ['eventing'], 'cpu_utilization': 2.820448249811131, 'mem_free': 3344236544, 'mem_total': 4201848832, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.243:8091 => {'services': ['kv'], 'cpu_utilization': 3.46309403437816, 'mem_free': 3223044096, 'mem_total': 4219445248, 'swap_mem_used': 156499968, 'swap_mem_total': 3758092288} 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['kv'], 'cpu_utilization': 4.222166373460668, 'mem_free': 2580516864, 'mem_total': 3705745408, 'swap_mem_used': 138481664, 'swap_mem_total': 3909087232} 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.247:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 3.447401774397972, 'mem_free': 3227222016, 'mem_total': 4219441152, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.249:8091 => {'services': ['kv'], 'cpu_utilization': 6.140573458513067, 'mem_free': 3203944448, 'mem_total': 4219445248, 'swap_mem_used': 36438016, 'swap_mem_total': 3758092288} 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:43:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:43:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:43:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:43:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:43:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:43:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:43:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:43:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:43:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:43:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:43:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:43:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:44:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-02-14 21:44:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.104.246 2021-02-14 21:44:29 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:44:29 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : ['metadata', 'src_bucket'] 2021-02-14 21:44:29 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 172.23.104.246 2021-02-14 21:44:30 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:44:30 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : ['src_bucket'] 2021-02-14 21:44:30 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 172.23.104.246 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : [] 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 172.23.104.246 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.243,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.249', 'ejectedNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.243,ns_1@172.23.104.247,ns_1@172.23.104.249', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:44:31 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:44:41 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-02-14 21:44:51 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 90.00 % 2021-02-14 21:45:11 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2021-02-14 21:45:11 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.104.246 port:8091 ssh_username:root ? [('ns_1@172.23.104.237', 8091), ('ns_1@172.23.104.243', 8091), ('ns_1@172.23.104.247', 8091), ('ns_1@172.23.104.249', 8091)] 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:45:21 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_eventing_rebalance_with_multiple_kv_nodes ============== 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:45:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:23 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:45:23 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:45:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:45:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:24 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:45:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:45:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:45:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:45:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:30 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 6170 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 6359 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:45:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:45:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:45:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:38 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:45:38 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:45:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:44 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 28428 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 28618 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:45:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:45:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:45:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:52 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:45:52 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:45:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:45:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:45:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:45:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:45:58 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 111771 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 111913 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:46:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:46:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:46:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:46:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:05 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:05 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:46:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:46:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:06 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:46:06 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:46:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:46:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:46:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:46:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:12 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 96542 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 96686 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:46:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:46:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:46:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:46:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:19 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:46:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:46:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:20 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:46:20 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:46:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:46:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:46:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:26 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 7953 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 8096 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:46:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:46:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:46:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:46:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:46:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:33 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:33 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:46:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:46:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:34 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:46:34 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:46:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:46:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:46:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:40 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 9218 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 9363 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:46:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:46:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:46:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:46:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:47 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:47 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:46:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:46:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:48 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:46:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:46:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:54 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:46:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:46:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:46:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 5026 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:46:59 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 5171 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:47:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:47:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:47:01 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:47:01 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:47:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:47:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:47:03 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:47:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:47:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:47:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:47:09 | INFO | MainProcess | test_thread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Couchbase server status: [' Process: 107992 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 108143 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2021-02-14 21:47:14 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-02-14 21:47:24 | INFO | MainProcess | Thread-657 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-659 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-656 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-658 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-661 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-664 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-666 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-668 | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:47:24 | INFO | MainProcess | Thread-656 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-659 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-661 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-664 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-658 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-668 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-666 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-657 | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:47:24 | INFO | MainProcess | Thread-656 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-657 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-659 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-658 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-668 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-664 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-661 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-666 | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:47:24 | INFO | MainProcess | Thread-656 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.246 2021-02-14 21:47:24 | INFO | MainProcess | Thread-656 | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: /opt/couchbase/bin/cbcollect_info 172.23.104.246-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-657 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.248 2021-02-14 21:47:24 | INFO | MainProcess | Thread-657 | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: /opt/couchbase/bin/cbcollect_info 172.23.104.248-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-659 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.237 2021-02-14 21:47:24 | INFO | MainProcess | Thread-659 | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: /opt/couchbase/bin/cbcollect_info 172.23.104.237-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-666 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.243 2021-02-14 21:47:24 | INFO | MainProcess | Thread-666 | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: /opt/couchbase/bin/cbcollect_info 172.23.104.243-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-668 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.245 2021-02-14 21:47:24 | INFO | MainProcess | Thread-668 | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: /opt/couchbase/bin/cbcollect_info 172.23.104.245-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-664 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.249 2021-02-14 21:47:24 | INFO | MainProcess | Thread-664 | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: /opt/couchbase/bin/cbcollect_info 172.23.104.249-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-661 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.104.247 2021-02-14 21:47:24 | INFO | MainProcess | Thread-661 | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: /opt/couchbase/bin/cbcollect_info 172.23.104.247-20210214-2147-diag.zip 2021-02-14 21:47:24 | INFO | MainProcess | Thread-658 | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 Collecting logs from 172.23.106.6 2021-02-14 21:47:24 | INFO | MainProcess | Thread-658 | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: /opt/couchbase/bin/cbcollect_info 172.23.106.6-20210214-2147-diag.zip 2021-02-14 21:48:50 | INFO | MainProcess | Thread-661 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.247 2021-02-14 21:48:50 | INFO | MainProcess | Thread-661 | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -f /root/172.23.104.247-20210214-2147-diag.zip 2021-02-14 21:48:50 | INFO | MainProcess | Thread-661 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:48:53 | INFO | MainProcess | Thread-668 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.245 2021-02-14 21:48:53 | INFO | MainProcess | Thread-668 | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -f /root/172.23.104.245-20210214-2147-diag.zip 2021-02-14 21:48:53 | INFO | MainProcess | Thread-668 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:48:57 | INFO | MainProcess | Thread-658 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.106.6 2021-02-14 21:48:57 | INFO | MainProcess | Thread-658 | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -f /root/172.23.106.6-20210214-2147-diag.zip 2021-02-14 21:48:57 | INFO | MainProcess | Thread-658 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:48:59 | INFO | MainProcess | Thread-666 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.243 2021-02-14 21:48:59 | INFO | MainProcess | Thread-666 | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -f /root/172.23.104.243-20210214-2147-diag.zip 2021-02-14 21:48:59 | INFO | MainProcess | Thread-666 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:49:00 | INFO | MainProcess | Thread-664 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.249 2021-02-14 21:49:01 | INFO | MainProcess | Thread-664 | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -f /root/172.23.104.249-20210214-2147-diag.zip 2021-02-14 21:49:01 | INFO | MainProcess | Thread-664 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:49:02 | INFO | MainProcess | Thread-659 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.237 2021-02-14 21:49:03 | INFO | MainProcess | Thread-659 | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -f /root/172.23.104.237-20210214-2147-diag.zip 2021-02-14 21:49:03 | INFO | MainProcess | Thread-659 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:49:12 | INFO | MainProcess | Thread-657 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.248 2021-02-14 21:49:13 | INFO | MainProcess | Thread-657 | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -f /root/172.23.104.248-20210214-2147-diag.zip 2021-02-14 21:49:13 | INFO | MainProcess | Thread-657 | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:49:21 | INFO | MainProcess | Thread-656 | [remote_util.execute_command_raw] command executed successfully with root Downloading zipped logs from 172.23.104.246 2021-02-14 21:49:22 | INFO | MainProcess | Thread-656 | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -f /root/172.23.104.246-20210214-2147-diag.zip 2021-02-14 21:49:22 | INFO | MainProcess | Thread-656 | [remote_util.execute_command_raw] command executed successfully with root ERROR ====================================================================== ERROR: test_eventing_rebalance_with_multiple_kv_nodes (eventing.eventing_rebalance.EventingRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/eventing/eventing_rebalance.py", line 1120, in test_eventing_rebalance_with_multiple_kv_nodes self.verify_eventing_results(self.function_name, self.docs_per_day * 2016, skip_stats_validation=True) File "./pytests/eventing/eventing_base.py", line 349, in verify_eventing_results timers_in_past,lcb)) Exception: missing data in destination bucket. Current : 10079 Expected : 10080 dcp_backlog : 0 TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.104.237': {'210': 1}} ---------------------------------------------------------------------- Ran 1 test in 1549.638s FAILED (errors=1) suite_tearDown (eventing.eventing_rebalance.EventingRebalance) ... summary so far suite eventing.eventing_rebalance.EventingRebalance , pass 0 , fail 1 failures so far... eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes testrunner logs, diags and results are available under /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers/logs/testrunner-21-Feb-14_21-13-01/test_1 *** Tests executed count: 1 Run after suite setup for eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [eventing_base.setUp] Starting Test: suite_tearDown 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.0.0-4454-enterprise 2021-02-14 21:49:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.get_nodes_versions] Node versions in cluster [] 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_tearDown============== 2021-02-14 21:49:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.246, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:49:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-02-14 21:49:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2021-02-14 21:49:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:49:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:49:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:49:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:49:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:49:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:49:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:49:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:49:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:49:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:49:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:49:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_tearDown ============== 2021-02-14 21:49:56 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.246, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_tearDown ============== 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:49:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:49:57 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:49:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:49:59 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:49:59 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:50:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:05 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 11062 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 11257 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:50:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:50:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:11 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:11 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:50:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:50:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:12 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:50:12 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:50:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:50:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:18 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 460 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 660 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:50:23 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:50:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:50:24 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:50:24 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:25 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:25 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:50:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:50:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:26 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:50:26 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:50:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:50:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:50:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:32 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 116190 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 116363 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:50:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:50:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:50:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:50:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:50:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:39 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:39 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:50:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:50:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:40 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:50:40 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:50:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:50:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:50:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:46 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 101366 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 101510 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:50:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:50:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:50:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:50:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:50:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:50:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:50:53 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:50:53 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:50:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:50:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:54 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:50:54 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:50:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:50:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:50:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:50:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:00 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 12556 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 12742 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:51:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:51:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:51:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:51:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:07 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:07 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:51:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:51:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:08 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:51:08 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:51:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:51:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:51:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:14 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 13847 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 13993 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:51:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:51:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:51:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:51:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:21 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:21 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:21 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:51:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:51:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:22 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:51:22 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:51:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:51:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:51:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:28 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 9671 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 9859 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:51:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:51:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:51:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:51:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:51:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:35 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:35 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:51:35 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:51:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:36 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:51:36 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:51:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:51:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:51:41 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:42 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 112719 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 112899 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:51:47 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2021-02-14 21:51:57 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2021-02-14 21:51:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:51:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:51:58 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:51:59 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:51:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:52:00 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:00 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:52:00 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:52:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:06 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 11781 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 11970 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:52:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:52:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:52:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:14 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:52:14 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:20 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 1221 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 1410 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:25 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:52:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:52:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:26 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:26 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:52:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:52:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:28 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:52:28 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:52:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:34 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 116945 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 117118 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:52:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:52:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:52:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:42 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:52:42 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:52:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:48 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 102130 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 102275 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:52:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:52:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:52:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:52:56 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:52:56 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:53:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:53:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:53:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:53:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:02 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 13308 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 13497 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:53:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:53:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:53:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:53:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:09 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:09 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:53:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:53:10 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:10 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:53:10 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:53:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:53:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:53:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:53:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:16 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 14612 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 14757 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:53:21 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:53:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:53:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:53:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:23 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:23 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:23 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:53:23 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:53:24 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:24 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:53:24 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:53:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:53:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:53:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:53:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:53:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:30 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:30 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:30 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:53:31 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:53:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:53:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:31 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 10477 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 10620 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:53:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:53:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:37 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:37 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:53:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:53:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:39 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:53:39 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:53:44 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:45 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 113540 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 113721 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:53:50 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.246 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '110', 'memoryTotal': 3705745408, 'memoryFree': 3220848640, 'mcdMemoryReserved': 2827, 'mcdMemoryAllocated': 2827, 'status': 'healthy', 'hostname': '172.23.104.246:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1454, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.246', 'ip': '172.23.104.246', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3534, 'curr_items': 0} 2021-02-14 21:54:01 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.246:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.104.246,8091,['kv']) 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.104.246: 8091:hostname=172.23.104.246&user=Administrator&password=password&services=kv 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.246:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.246:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.246:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.248 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '99', 'memoryTotal': 3705745408, 'memoryFree': 3162873856, 'mcdMemoryReserved': 2827, 'mcdMemoryAllocated': 2827, 'status': 'healthy', 'hostname': '172.23.104.248:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1454, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.248', 'ip': '172.23.104.248', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3534, 'curr_items': 0} 2021-02-14 21:54:01 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.248:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1894 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.248:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:54:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.248:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.248:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.6 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '84', 'memoryTotal': 4201840640, 'memoryFree': 3620630528, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.106.6:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.106.6', 'ip': '172.23.106.6', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-14 21:54:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.6:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.6:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:54:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.6:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.6:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.237 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '70', 'memoryTotal': 4201848832, 'memoryFree': 3656597504, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.104.237:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.237', 'ip': '172.23.104.237', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} 2021-02-14 21:54:03 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.237:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.237:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:54:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.247 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '55', 'memoryTotal': 4219441152, 'memoryFree': 3667853312, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.247:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.247', 'ip': '172.23.104.247', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:54:04 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.247:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.247:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:54:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.249 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '45', 'memoryTotal': 4219445248, 'memoryFree': 3675598848, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.249:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.249', 'ip': '172.23.104.249', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:54:05 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.249:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.249:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.243 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '30', 'memoryTotal': 4219445248, 'memoryFree': 3707387904, 'mcdMemoryReserved': 3219, 'mcdMemoryAllocated': 3219, 'status': 'healthy', 'hostname': '172.23.104.243:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1748, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.243', 'ip': '172.23.104.243', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4023, 'curr_items': 0} 2021-02-14 21:54:05 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.243:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.243:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.243:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.243:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.245 port:8091 ssh_username:root, nodes/self 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '19', 'memoryTotal': 25084071936, 'memoryFree': 24289914880, 'mcdMemoryReserved': 19137, 'mcdMemoryAllocated': 19137, 'status': 'healthy', 'hostname': '172.23.104.245:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 13687, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.245', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 23922, 'curr_items': 0} 2021-02-14 21:54:06 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.104.245:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=12821 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.245:8091:port=8091&username=Administrator&password=password 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:54:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.245:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.245:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). 2021-02-14 21:54:07 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:54:07 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.246 **** 2021-02-14 21:54:07 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.104.246:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2021-02-14 21:54:07 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2021-02-14 21:54:07 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:54:12 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2021-02-14 21:54:12 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2021-02-14 21:54:13 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.248:8091 to cluster 2021-02-14 21:54:13 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.248:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:54:15 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.6:8091 to cluster 2021-02-14 21:54:15 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.6:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:54:18 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.237:8091 to cluster 2021-02-14 21:54:18 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.237:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:54:20 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.104.247:8091 to cluster 2021-02-14 21:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.104.247:8091 to this cluster @172.23.104.246:8091 2021-02-14 21:54:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:54:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:54:23 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:54:23 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2021-02-14 21:54:33 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2021-02-14 21:54:33 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.02762246131897 sec 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.246 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_tearDown ============== 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:54:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:54:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:54:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:54:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:54:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:54:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:54:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:54:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:54:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:54:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:54:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:54:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:54:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.237:8091 => {'services': ['eventing'], 'cpu_utilization': 8.614137319483152, 'mem_free': 3630080000, 'mem_total': 4201848832, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['kv'], 'cpu_utilization': 0.9542943244600703, 'mem_free': 3180830720, 'mem_total': 3705745408, 'swap_mem_used': 83517440, 'swap_mem_total': 3909087232} 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.247:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 11.02402022756005, 'mem_free': 3592138752, 'mem_total': 4219441152, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.248:8091 => {'services': ['kv'], 'cpu_utilization': 0.7010515773660491, 'mem_free': 3134803968, 'mem_total': 3705745408, 'swap_mem_used': 0, 'swap_mem_total': 3909087232} 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.6:8091 => {'services': ['kv'], 'cpu_utilization': 0.77850326469111, 'mem_free': 3593678848, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:55:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2021-02-14 21:55:10 | INFO | MainProcess | MainThread | [eventing_rebalance.setUp] 100 2021-02-14 21:55:10 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:55:10 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket src_bucket 2021-02-14 21:55:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:11 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:55:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:12 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:12 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:13 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:13 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:14 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:14 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:15 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 src_bucket 2021-02-14 21:55:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 src_bucket 2021-02-14 21:55:16 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:55:17 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:55:17 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2021-02-14 21:55:17 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:18 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:18 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:19 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:19 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:20 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:20 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:21 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:21 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:55:21 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.104.246 to accept set ops 2021-02-14 21:55:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 dst_bucket 2021-02-14 21:55:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 dst_bucket 2021-02-14 21:55:22 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 dst_bucket 2021-02-14 21:55:22 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.246:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2021-02-14 21:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket metadata 2021-02-14 21:55:23 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:55:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:25 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:26 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:26 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2021-02-14 21:55:26 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:27 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:55:27 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:28 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.104.246 to accept set ops 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.246:11210 metadata 2021-02-14 21:55:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.248:11210 metadata 2021-02-14 21:55:29 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.6:11210 metadata 2021-02-14 21:55:29 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2021-02-14 21:55:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:55:29 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:55:29 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:55:30 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 468.588238ms 2021-02-14 21:55:30 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI 2021-02-14 21:55:30 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.782857667s 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 25.306019ms 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.384168ms 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI 2021-02-14 21:55:32 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.833671303s 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 19.054592ms 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.161068ms 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI 2021-02-14 21:55:33 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.746789874s 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary' 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.968136ms 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.104.237 is [] 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.104.237 port:8091 ssh_username:root] 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:55:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:55:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:36 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 172.23.104.237:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2021-02-14 21:55:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.104.247 port:8091 ssh_username:root] 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=select+RAW%28count%28%2A%29%29+from+metadata 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [eventing_base.tearDown] number of documents in metadata bucket 0 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.237:8091 => {'services': ['eventing'], 'cpu_utilization': 2.236180904522613, 'mem_free': 3585314816, 'mem_total': 4201848832, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.246:8091 => {'services': ['kv'], 'cpu_utilization': 16.8281489036206, 'mem_free': 3008516096, 'mem_total': 3705745408, 'swap_mem_used': 83513344, 'swap_mem_total': 3909087232} 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.247:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 14.39857288481142, 'mem_free': 3316903936, 'mem_total': 4219441152, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.248:8091 => {'services': ['kv'], 'cpu_utilization': 14.40892947742263, 'mem_free': 2990759936, 'mem_total': 3705745408, 'swap_mem_used': 0, 'swap_mem_total': 3909087232} 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.6:8091 => {'services': ['kv'], 'cpu_utilization': 14.95518565941101, 'mem_free': 3446267904, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:55:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:55:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:55:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:55:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:55:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:55:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:55:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:55:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:55:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:55:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:55:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:56:11 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_tearDown ============== 2021-02-14 21:56:11 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.104.246 2021-02-14 21:56:12 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:56:12 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : ['metadata', 'src_bucket'] 2021-02-14 21:56:12 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : dst_bucket from 172.23.104.246 2021-02-14 21:56:13 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:56:13 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : ['src_bucket'] 2021-02-14 21:56:13 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : metadata from 172.23.104.246 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.246 existing buckets : [] 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : src_bucket from 172.23.104.246 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.248, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.6, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.237, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.247, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.249, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.243, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.245, nothing to delete 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.246,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'ejectedNodes': 'ns_1@172.23.104.237,ns_1@172.23.104.247,ns_1@172.23.104.248,ns_1@172.23.106.6', 'user': 'Administrator', 'password': 'password'} 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2021-02-14 21:56:14 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2021-02-14 21:56:24 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-02-14 21:56:34 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 70.00 % 2021-02-14 21:56:54 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2021-02-14 21:56:54 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.104.246 port:8091 ssh_username:root ? [('ns_1@172.23.104.237', 8091), ('ns_1@172.23.104.247', 8091), ('ns_1@172.23.104.248', 8091), ('ns_1@172.23.106.6', 8091)] 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.246:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.246:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.248:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.248:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.6:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.6:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.237:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.237:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.247:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.247:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.249:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.249:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.243:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.243:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.245:8091 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.245:8091 is running 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_tearDown ============== 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:57:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:57:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:05 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:05 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:57:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl stop couchbase-server.service 2021-02-14 21:57:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:07 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:57:07 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.246 with username:root, attempt#1 of 5 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.246 as root 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.246 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:57:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl start couchbase-server.service 2021-02-14 21:57:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:13 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.246:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.246: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 16932 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.246: with pid 17121 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:57:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:19 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:19 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:19 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:57:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl stop couchbase-server.service 2021-02-14 21:57:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:20 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:57:20 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:57:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:57:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:25 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.248 with username:root, attempt#1 of 5 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.248 as root 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.248 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl start couchbase-server.service 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:26 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.248:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.248: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 5939 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.248: with pid 6129 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:57:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:57:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:57:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:57:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:33 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:33 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:57:33 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl stop couchbase-server.service 2021-02-14 21:57:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:34 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:57:34 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:57:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:57:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:39 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.6 as root 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.106.6 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl start couchbase-server.service 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:40 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.106.6:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.6: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 121651 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.106.6: with pid 121793 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:57:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:57:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:57:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:57:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:57:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:47 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:47 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:47 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:57:47 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl stop couchbase-server.service 2021-02-14 21:57:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:48 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:57:48 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:57:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:57:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:57:53 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.237 with username:root, attempt#1 of 5 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.237 as root 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.237 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl start couchbase-server.service 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:54 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.237:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.237: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 107552 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.237: with pid 107727 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:57:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:58:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:58:00 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:58:00 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:01 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:01 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:58:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl stop couchbase-server.service 2021-02-14 21:58:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:02 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:58:02 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:58:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:58:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:58:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.247 with username:root, attempt#1 of 5 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.247 as root 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.247 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl start couchbase-server.service 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:08 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.247:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 18298 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.247: with pid 18440 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:58:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:58:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:58:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:58:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:58:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:15 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:15 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:58:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl stop couchbase-server.service 2021-02-14 21:58:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:17 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:58:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.249 with username:root, attempt#1 of 5 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.249 as root 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.249 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl start couchbase-server.service 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:22 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.249:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:58:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:58:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:27 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 19002 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:58:27 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.249: with pid 19146 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:58:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:58:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:29 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:29 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:58:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl stop couchbase-server.service 2021-02-14 21:58:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:31 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:58:31 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.243 with username:root, attempt#1 of 5 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.243 as root 2021-02-14 21:58:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.243 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl start couchbase-server.service 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:37 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.243:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.243: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 14850 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.243: with pid 14995 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:58:43 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:58:43 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:43 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:43 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:43 | INFO | MainProcess | MainThread | [remote_util.stop_couchbase] Running systemd command on this server 2021-02-14 21:58:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl stop couchbase-server.service 2021-02-14 21:58:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:45 | INFO | MainProcess | MainThread | [basetestcase.stop_server] Couchbase stopped 2021-02-14 21:58:45 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.245 with username:root, attempt#1 of 5 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.245 as root 2021-02-14 21:58:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.is_couchbase_installed] 172.23.104.245 **** The linux version file /opt/couchbase/ VERSION.txt exists 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Starting couchbase server 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Running systemd command on this server 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl start couchbase-server.service 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:51 | INFO | MainProcess | MainThread | [remote_util.sleep] 172.23.104.245:sleep for 5 secs. waiting for couchbase server to come up ... 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.245: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with root 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [remote_util.start_couchbase] Couchbase server status: [' Process: 117710 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [remote_util.is_process_running] Checking for process beam.smp on linux 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [remote_util.is_process_running] process beam.smp is running on 172.23.104.245: with pid 117860 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [basetestcase.start_server] Couchbase started 2021-02-14 21:58:56 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 10 secs. ... ok ---------------------------------------------------------------------- Ran 1 test in 584.277s OK Cluster instance shutdown with force During the test, Remote Connections: 0, Disconnections: 1161 ('Thread', , 'was not properly terminated, will be terminated now.') Shutting down the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... ('Thread', , 'was not properly terminated, will be terminated now.') Stopping the thread... *** TestRunner *** workspace is /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers fails is 1 1 Desc1: 7.0.0-4454 - eventing rebalance_bucket_op_with_timers - centos (0/1) python3 scripts/rerun_jobs.py 7.0.0-4454 --executor_jenkins_job --run_params=get-cbcollect-info=True,GROUP=bucket_op_with_timers INFO:merge_reports:Merging of report files from ['Old_Report_mergedreport-21-Feb-13_17-53-34-eventing.eventing_rebalance.EventingRebalance.xml', 'logs/**/*.xml'] INFO:merge_reports:-- Old_Report_mergedreport-21-Feb-13_17-53-34-eventing.eventing_rebalance.EventingRebalance.xml -- INFO:merge_reports:-- logs/testrunner-21-Feb-14_21-13-01/report-21-Feb-14_21-13-01-eventing.eventing_rebalance.EventingRebalance.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) eventing.eventing_rebalance.EventingRebalance, Number of Tests=8 INFO:merge_reports:Summary file is at /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_with_timers/logs/testrunner-21-Feb-14_21-59-07/merged_summary/mergedreport-21-Feb-14_21-59-07-eventing.eventing_rebalance.EventingRebalance.xml Merging xmls summary so far suite eventing.eventing_rebalance.EventingRebalance , pass 7 , fail 1 failures so far... eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes merged xmls No more failed tests. Stopping reruns [description-setter] Description set: 7.0.0-4454 - eventing rebalance_bucket_op_with_timers - centos (0/1) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) #314952 is still in progress; ignoring for purposes of comparison #314952 is still in progress; ignoring for purposes of comparison Sending email to: balakumaran.gopal@couchbase.com mihir.kamdar@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Finished: UNSTABLE