Started by remote host 172.23.109.52 [EnvInject] - Loading node environment variables. Building remotely on slv-s82202 (P0 aws_analytics) in workspace /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [centosnonroot-p0-os_certify-vset00-00-nsserver] $ /bin/sh -xe /tmp/jenkins4696134942472982645.sh ++ echo os_certify-nsserver-Mar-01-03:07:42-7.1.0-2407 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='os_certify nsserver' + echo Desc: 7.1.0-2407 - os_certify nsserver - centosnonroot Desc: 7.1.0-2407 - os_certify nsserver - centosnonroot + echo newState=available + newState=available Success build forhudson.tasks.Shell@5ee7b57d [description-setter] Description set: 7.1.0-2407 - os_certify nsserver - centosnonroot Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@58d0b419 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@72813b5f Cloning the remote Git repository Using shallow clone Cloning repository https://github.com/couchbase/testrunner > /usr/bin/git init /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision cdebbe6f2180879eba78f1ed24a9f6e1c87e4299 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f cdebbe6f2180879eba78f1ed24a9f6e1c87e4299 > /usr/bin/git rev-list cdebbe6f2180879eba78f1ed24a9f6e1c87e4299 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #447681 jenkins-test_suite_executor-447681 # timeout=10 No emails were triggered. [centosnonroot-p0-os_certify-vset00-00-nsserver] $ /bin/sh -xe /tmp/jenkins9181531976346669597.sh + echo Desc: os_certify-nsserver-Mar-01-03:07:42-7.1.0-2407 Desc: os_certify-nsserver-Mar-01-03:07:42-7.1.0-2407 [description-setter] Could not determine description. [centosnonroot-p0-os_certify-vset00-00-nsserver] $ /bin/sh -xe /tmp/jenkins1587750522902237651.sh [centosnonroot-p0-os_certify-vset00-00-nsserver] $ /bin/sh -xe /tmp/jenkins5775638734180778201.sh + py_executable=python2 + echo 7.1.0-2407 + grep -E '7\.0|7\.1|0\.0' 7.1.0-2407 + py_executable=python3 + [[ 7.1.0-2407 > 6.5 ]] + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + rerun_job=true + touch rerun_props_file + '[' true == false ']' [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centosnonroot-p0-os_certify-vset00-00-nsserver] $ /bin/bash /tmp/jenkins7847221629649757877.sh 7.1.0-2407 Already on 'master' From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. Set ALLOW_HTP to False so test could run. HEAD is now at cdebbe6 Fixes for query context collections job. the major release is 7 "172.23.106.52","172.23.105.126","172.23.105.158","172.23.104.172","172.23.105.162","172.23.105.127" Searching for httplib2 Best match: httplib2 0.17.0 Adding httplib2 0.17.0 to easy-install.pth file Using /usr/local/lib/python3.7/site-packages Processing dependencies for httplib2 Finished processing dependencies for httplib2 centos Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.ocf.berkeley.edu * extras: linux.mirrors.es.net * updates: mirror.hostduplex.com Package 2:docker-1.13.1-208.git7d71120.el7_9.x86_64 already installed and latest version Nothing to do Using default tag: latest Trying to pull repository docker.io/jamesdbloom/mockserver ... latest: Pulling from docker.io/jamesdbloom/mockserver Digest: sha256:bc41010b9e920f5e1b75a226e8eda05eb4eb9843dad042e9e3950d1b7de823f4 Status: Image is up to date for docker.io/jamesdbloom/mockserver:latest [global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [cbbackupmgr] staging_directory:/tmp/cbbackupmgr-staging [cluster1] 1:_3 2:_4 [cluster2] 1:_5 2:_6 [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:dynamic services:kv,index,n1ql,fts [_2] ip:dynamic services:kv,index,n1ql,fts [_3] ip:dynamic services:kv,index,n1ql,fts [_4] ip:dynamic services:kv,index,n1ql,fts [_5] ip:dynamic services:kv,index,n1ql,fts [_6] ip:dynamic services:kv,index,n1ql,fts python3 scripts/populateIni.py -s "172.23.106.52","172.23.105.126","172.23.105.158","172.23.104.172","172.23.105.162","172.23.105.127" -d None -a None -i /tmp/testexec_reformat.30028.ini -p centosnonroot -o /tmp/testexec.30028.ini -k {"username":"nonroot","password":"couchbase"} INFO:root:SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.105.158 as root INFO:root:SSH Connected to 172.23.105.127 as root INFO:root:SSH Connected to 172.23.105.126 as root INFO:root:SSH Connected to 172.23.105.162 as root INFO:root:SSH Connected to 172.23.104.172 as root INFO:root:SSH Connected to 172.23.106.52 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.126: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.127: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.172: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.158: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.162: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.52: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root in main the ini file is /tmp/testexec_reformat.30028.ini the given server info is "172.23.106.52","172.23.105.126","172.23.105.158","172.23.104.172","172.23.105.162","172.23.105.127" Collecting memory info from 172.23.105.126 Collecting memory info from 172.23.105.127 Collecting memory info from 172.23.104.172 Collecting memory info from 172.23.105.158 Collecting memory info from 172.23.105.162 Collecting memory info from 172.23.106.52 the servers memory info is [('172.23.104.172', 3854856), ('172.23.106.52', 3854856), ('172.23.105.158', 4103152), ('172.23.105.126', 4103152), ('172.23.105.127', 4103152), ('172.23.105.162', 4103152)] [global] username:nonroot password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [cbbackupmgr] staging_directory:/tmp/cbbackupmgr-staging [cluster1] 1:_3 2:_4 [cluster2] 1:_5 2:_6 [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.104.172 services:kv,index,n1ql,fts [_2] ip:172.23.106.52 services:kv,index,n1ql,fts [_3] ip:172.23.105.158 services:kv,index,n1ql,fts [_4] ip:172.23.105.126 services:kv,index,n1ql,fts [_5] ip:172.23.105.127 services:kv,index,n1ql,fts [_6] ip:172.23.105.162 services:kv,index,n1ql,fts extra install is ,timeout=1200,skip_local_download=True Local time: Tue 2022-03-01 03:08:00 PST Universal time: Tue 2022-03-01 11:08:00 UTC RTC time: Tue 2022-03-01 11:07:59 Time zone: US/Pacific (PST, -0800) NTP enabled: no NTP synchronized: yes RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2021-11-07 01:59:59 PDT Sun 2021-11-07 01:00:00 PST Next DST change: DST begins (the clock jumps one hour forward) at Sun 2022-03-13 01:59:59 PST Sun 2022-03-13 03:00:00 PDT python3 scripts/ssh.py -i /tmp/testexec_root.30028.ini iptables -F INFO:root:SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.105.126 as root INFO:root:SSH Connected to 172.23.105.158 as root INFO:root:SSH Connected to 172.23.105.127 as root INFO:root:SSH Connected to 172.23.105.162 as root INFO:root:SSH Connected to 172.23.106.52 as root INFO:root:SSH Connected to 172.23.104.172 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.127: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.162: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.158: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.126: iptables -F INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.52: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.172: iptables -F INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:command executed successfully with root 172.23.105.162 172.23.105.127 172.23.105.126 172.23.105.158 172.23.106.52 172.23.104.172 Initial version: python3 scripts/new_install.py -i /tmp/testexec.30028.ini -p timeout=1800,skip_local_download=False,get-cbcollect-info=True,version=7.1.0-2407,product=cb,debug_logs=True,ntp=True,url=,timeout=1200,skip_local_download=True 2022-03-01 03:08:02,016 - root - WARNING - URL: is not valid, will use version to locate build 2022-03-01 03:08:02,017 - root - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:02,154 - root - INFO - SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:08:02,423 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:02,806 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:02,809 - root - INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:02,948 - root - INFO - SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:08:03,218 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:03,592 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:03,595 - root - INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:03,697 - root - INFO - SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:08:04,001 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:04,343 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:04,346 - root - INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:04,447 - root - INFO - SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:08:04,754 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:05,086 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:05,089 - root - INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:05,194 - root - INFO - SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:08:05,482 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:05,827 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:05,832 - root - INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:05,938 - root - INFO - SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:08:06,261 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:06,595 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:06,598 - root - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:06,737 - root - INFO - SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:08:07,006 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:07,387 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:07,390 - root - INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:07,524 - root - INFO - SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:08:07,797 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:08,182 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:08,185 - root - INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:08,287 - root - INFO - SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:08:08,605 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:08,942 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:08,945 - root - INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:09,051 - root - INFO - SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:08:09,358 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:09,701 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:09,704 - root - INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:09,807 - root - INFO - SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:08:10,092 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:10,422 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:10,424 - root - INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:10,537 - root - INFO - SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:08:10,850 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:11,184 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:11,185 - root - INFO - Check if ntp is installed 2022-03-01 03:08:11,186 - root - INFO - This OS version centos 7 2022-03-01 03:08:11,186 - root - INFO - running command.raw on 172.23.104.172: systemctl status ntpd 2022-03-01 03:08:11,187 - root - INFO - Check if ntp is installed 2022-03-01 03:08:11,187 - root - INFO - Check if ntp is installed 2022-03-01 03:08:11,188 - root - INFO - Check if ntp is installed 2022-03-01 03:08:11,188 - root - INFO - This OS version centos 7 2022-03-01 03:08:11,188 - root - INFO - This OS version centos 7 2022-03-01 03:08:11,189 - root - INFO - Check if ntp is installed 2022-03-01 03:08:11,189 - root - INFO - This OS version centos 7 2022-03-01 03:08:11,189 - root - INFO - running command.raw on 172.23.106.52: systemctl status ntpd 2022-03-01 03:08:11,189 - root - INFO - Check if ntp is installed 2022-03-01 03:08:11,190 - root - INFO - running command.raw on 172.23.105.158: systemctl status ntpd 2022-03-01 03:08:11,190 - root - INFO - This OS version centos 7 2022-03-01 03:08:11,190 - root - INFO - running command.raw on 172.23.105.126: systemctl status ntpd 2022-03-01 03:08:11,191 - root - INFO - This OS version centos 7 2022-03-01 03:08:11,191 - root - INFO - running command.raw on 172.23.105.127: systemctl status ntpd 2022-03-01 03:08:11,192 - root - INFO - running command.raw on 172.23.105.162: systemctl status ntpd 2022-03-01 03:08:11,224 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,224 - root - INFO - running command.raw on 172.23.105.158: ntpstat 2022-03-01 03:08:11,227 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,228 - root - INFO - running command.raw on 172.23.105.127: ntpstat 2022-03-01 03:08:11,229 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,229 - root - INFO - running command.raw on 172.23.105.126: ntpstat 2022-03-01 03:08:11,231 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,231 - root - INFO - running command.raw on 172.23.104.172: ntpstat 2022-03-01 03:08:11,234 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,235 - root - INFO - running command.raw on 172.23.106.52: ntpstat 2022-03-01 03:08:11,256 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,256 - root - INFO - running command.raw on 172.23.105.162: ntpstat 2022-03-01 03:08:11,302 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,303 - root - INFO - running command.raw on 172.23.104.172: timedatectl status 2022-03-01 03:08:11,331 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,331 - root - INFO - running command.raw on 172.23.105.158: timedatectl status 2022-03-01 03:08:11,334 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,335 - root - INFO - running command.raw on 172.23.105.126: timedatectl status 2022-03-01 03:08:11,337 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,337 - root - INFO - running command.raw on 172.23.105.127: timedatectl status 2022-03-01 03:08:11,366 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,366 - root - INFO - running command.raw on 172.23.105.162: timedatectl status 2022-03-01 03:08:11,379 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,379 - root - INFO - running command.raw on 172.23.106.52: timedatectl status 2022-03-01 03:08:11,398 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,398 - root - INFO - running command.raw on 172.23.105.126: date 2022-03-01 03:08:11,400 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,400 - root - INFO - running command.raw on 172.23.105.127: date 2022-03-01 03:08:11,430 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,430 - root - INFO - running command.raw on 172.23.104.172: date 2022-03-01 03:08:11,444 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,445 - root - INFO - running command.raw on 172.23.105.158: date 2022-03-01 03:08:11,460 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,462 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,466 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,467 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,468 - root - INFO - running command.raw on 172.23.105.162: date 2022-03-01 03:08:11,479 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,479 - root - INFO - running command.raw on 172.23.106.52: date 2022-03-01 03:08:11,487 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,498 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,521 - root - INFO - ['Tue Mar 1 03:08:11 PST 2022'] IP: 172.23.105.126 2022-03-01 03:08:11,522 - root - INFO - ['Tue Mar 1 03:08:11 PST 2022'] IP: 172.23.105.127 2022-03-01 03:08:11,526 - root - INFO - ['Tue Mar 1 03:08:11 PST 2022'] IP: 172.23.105.158 2022-03-01 03:08:11,545 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:11,557 - root - INFO - ['Tue Mar 1 03:08:11 PST 2022'] IP: 172.23.105.162 2022-03-01 03:08:11,563 - root - INFO - ['Tue Mar 1 03:08:11 PST 2022'] IP: 172.23.104.172 2022-03-01 03:08:11,613 - root - INFO - ['Tue Mar 1 03:08:11 PST 2022'] IP: 172.23.106.52 2022-03-01 03:08:11,613 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:11,623 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm is live 2022-03-01 03:08:11,623 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:11,625 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm is live 2022-03-01 03:08:11,625 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:11,626 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm is live 2022-03-01 03:08:11,626 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:11,627 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm is live 2022-03-01 03:08:11,628 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:11,629 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm is live 2022-03-01 03:08:11,629 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:11,630 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm is live 2022-03-01 03:08:11,632 - root - INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 2022-03-01 03:08:11,634 - root - INFO - SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 2022-03-01 03:08:11,639 - root - INFO - SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 2022-03-01 03:08:11,641 - root - INFO - SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 2022-03-01 03:08:11,643 - root - INFO - SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 2022-03-01 03:08:11,645 - root - INFO - SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 2022-03-01 03:08:11,749 - root - INFO - SSH Connected to 172.23.105.127 as root 2022-03-01 03:08:11,753 - root - INFO - SSH Connected to 172.23.105.126 as root 2022-03-01 03:08:11,755 - root - INFO - SSH Connected to 172.23.105.162 as root 2022-03-01 03:08:11,757 - root - INFO - SSH Connected to 172.23.105.158 as root 2022-03-01 03:08:11,777 - root - INFO - SSH Connected to 172.23.104.172 as root 2022-03-01 03:08:11,778 - root - INFO - SSH Connected to 172.23.106.52 as root 2022-03-01 03:08:12,042 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:12,050 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:12,063 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:12,082 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:12,088 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:12,090 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:12,410 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:12,411 - root - INFO - running command.raw on 172.23.105.162: umount -a -t nfs,nfs4 -f -l;/home/nonroot/opt/couchbase/bin/couchbase-server -k; kill -9 `ps -ef |egrep couchbase|cut -f3 -d' '`; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:12,412 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:12,414 - root - INFO - running command.raw on 172.23.105.126: umount -a -t nfs,nfs4 -f -l;/home/nonroot/opt/couchbase/bin/couchbase-server -k; kill -9 `ps -ef |egrep couchbase|cut -f3 -d' '`; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:12,416 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:12,417 - root - INFO - running command.raw on 172.23.105.158: umount -a -t nfs,nfs4 -f -l;/home/nonroot/opt/couchbase/bin/couchbase-server -k; kill -9 `ps -ef |egrep couchbase|cut -f3 -d' '`; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:12,423 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:12,423 - root - INFO - running command.raw on 172.23.105.127: umount -a -t nfs,nfs4 -f -l;/home/nonroot/opt/couchbase/bin/couchbase-server -k; kill -9 `ps -ef |egrep couchbase|cut -f3 -d' '`; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:12,437 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:12,438 - root - INFO - running command.raw on 172.23.106.52: umount -a -t nfs,nfs4 -f -l;/home/nonroot/opt/couchbase/bin/couchbase-server -k; kill -9 `ps -ef |egrep couchbase|cut -f3 -d' '`; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:12,447 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:12,448 - root - INFO - running command.raw on 172.23.104.172: umount -a -t nfs,nfs4 -f -l;/home/nonroot/opt/couchbase/bin/couchbase-server -k; kill -9 `ps -ef |egrep couchbase|cut -f3 -d' '`; rm -rf /home/nonroot/opt/couchbase/ > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:14,546 - root - INFO - command executed successfully with root 2022-03-01 03:08:14,607 - root - INFO - command executed with root but got an error ['kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]'] ... 2022-03-01 03:08:14,651 - root - INFO - command executed successfully with root 2022-03-01 03:08:14,826 - root - INFO - command executed with root but got an error ['kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]'] ... 2022-03-01 03:08:15,572 - root - INFO - command executed with root but got an error ['kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]'] ... 2022-03-01 03:08:15,588 - root - INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:15,620 - root - INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:15,691 - root - INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:15,695 - root - INFO - SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:08:15,703 - root - INFO - command executed with root but got an error ['kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]'] ... 2022-03-01 03:08:15,729 - root - INFO - SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:08:15,798 - root - INFO - SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:08:15,836 - root - INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:15,942 - root - INFO - SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:08:16,010 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:16,022 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:16,119 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:16,259 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:16,350 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:16,350 - root - INFO - Done with uninstall on 172.23.105.126. 2022-03-01 03:08:16,361 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:16,361 - root - INFO - Done with uninstall on 172.23.105.127. 2022-03-01 03:08:16,459 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:16,460 - root - INFO - Done with uninstall on 172.23.105.158. 2022-03-01 03:08:16,599 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:16,600 - root - INFO - Done with uninstall on 172.23.105.162. 2022-03-01 03:08:16,832 - root - INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:16,870 - root - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:08:16,968 - root - INFO - SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:08:17,009 - root - INFO - SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:08:17,247 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:17,288 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:08:17,631 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:17,632 - root - INFO - Done with uninstall on 172.23.106.52. 2022-03-01 03:08:17,665 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:08:17,665 - root - INFO - Done with uninstall on 172.23.104.172. 2022-03-01 03:08:21,650 - root - INFO - running command.raw on 172.23.104.172: cd /home/nonroot/; wget -Nq http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:25,496 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:25,499 - root - INFO - running command.raw on 172.23.104.172: cd /home/nonroot/ && wc -c couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:25,575 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:25,575 - root - INFO - running command.raw on 172.23.104.172: ls -lh /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:25,643 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:25,643 - root - INFO - running command.raw on 172.23.106.52: cd /home/nonroot/; wget -Nq http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:29,441 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:29,443 - root - INFO - running command.raw on 172.23.106.52: cd /home/nonroot/ && wc -c couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:29,522 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:29,522 - root - INFO - running command.raw on 172.23.106.52: ls -lh /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:29,589 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:29,589 - root - INFO - running command.raw on 172.23.105.158: cd /home/nonroot/; wget -Nq http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:30,664 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:30,667 - root - INFO - running command.raw on 172.23.105.158: cd /home/nonroot/ && wc -c couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:30,693 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:30,693 - root - INFO - running command.raw on 172.23.105.158: ls -lh /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:30,753 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:30,753 - root - INFO - running command.raw on 172.23.105.126: cd /home/nonroot/; wget -Nq http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:31,655 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:31,657 - root - INFO - running command.raw on 172.23.105.126: cd /home/nonroot/ && wc -c couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:31,677 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:31,677 - root - INFO - running command.raw on 172.23.105.126: ls -lh /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:31,738 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:31,738 - root - INFO - running command.raw on 172.23.105.127: cd /home/nonroot/; wget -Nq http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:32,713 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:32,715 - root - INFO - running command.raw on 172.23.105.127: cd /home/nonroot/ && wc -c couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:32,742 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:32,743 - root - INFO - running command.raw on 172.23.105.127: ls -lh /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:32,803 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:32,803 - root - INFO - running command.raw on 172.23.105.162: cd /home/nonroot/; wget -Nq http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2407/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:33,712 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:33,714 - root - INFO - running command.raw on 172.23.105.162: cd /home/nonroot/ && wc -c couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:33,731 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:33,732 - root - INFO - running command.raw on 172.23.105.162: ls -lh /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm 2022-03-01 03:08:33,791 - root - INFO - command executed successfully with nonroot 2022-03-01 03:08:33,792 - root - INFO - running command.raw on 172.23.104.172: ls -l /home/nonroot/opt/couchbase/bin/ 2022-03-01 03:08:33,793 - root - INFO - running command.raw on 172.23.106.52: ls -l /home/nonroot/opt/couchbase/bin/ 2022-03-01 03:08:33,794 - root - INFO - running command.raw on 172.23.105.158: ls -l /home/nonroot/opt/couchbase/bin/ 2022-03-01 03:08:33,795 - root - INFO - running command.raw on 172.23.105.126: ls -l /home/nonroot/opt/couchbase/bin/ 2022-03-01 03:08:33,797 - root - INFO - running command.raw on 172.23.105.127: ls -l /home/nonroot/opt/couchbase/bin/ 2022-03-01 03:08:33,798 - root - INFO - running command.raw on 172.23.105.162: ls -l /home/nonroot/opt/couchbase/bin/ 2022-03-01 03:08:33,815 - root - INFO - command executed with nonroot but got an error ['ls: cannot access /home/nonroot/opt/couchbase/bin/: No such file or directory'] ... 2022-03-01 03:08:33,815 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.105.158.. 2022-03-01 03:08:33,819 - root - INFO - command executed with nonroot but got an error ['ls: cannot access /home/nonroot/opt/couchbase/bin/: No such file or directory'] ... 2022-03-01 03:08:33,820 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.105.126.. 2022-03-01 03:08:33,821 - root - INFO - command executed with nonroot but got an error ['ls: cannot access /home/nonroot/opt/couchbase/bin/: No such file or directory'] ... 2022-03-01 03:08:33,821 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.105.127.. 2022-03-01 03:08:33,822 - root - INFO - command executed with nonroot but got an error ['ls: cannot access /home/nonroot/opt/couchbase/bin/: No such file or directory'] ... 2022-03-01 03:08:33,822 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.106.52.. 2022-03-01 03:08:33,824 - root - INFO - command executed with nonroot but got an error ['ls: cannot access /home/nonroot/opt/couchbase/bin/: No such file or directory'] ... 2022-03-01 03:08:33,824 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.104.172.. 2022-03-01 03:08:33,852 - root - INFO - command executed with nonroot but got an error ['ls: cannot access /home/nonroot/opt/couchbase/bin/: No such file or directory'] ... 2022-03-01 03:08:33,852 - root - INFO - Waiting 20s to remove previous yum repo on 172.23.105.162.. 2022-03-01 03:08:53,835 - root - INFO - running command.raw on 172.23.105.158: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:53,840 - root - INFO - running command.raw on 172.23.105.126: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:53,841 - root - INFO - running command.raw on 172.23.106.52: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:53,841 - root - INFO - running command.raw on 172.23.104.172: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:53,842 - root - INFO - running command.raw on 172.23.105.127: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:08:53,853 - root - INFO - running command.raw on 172.23.105.162: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:09:41,194 - root - INFO - command executed with nonroot but got an error ['2445061 blocks'] ... 2022-03-01 03:09:41,194 - root - INFO - Waiting 20s for install to complete on 172.23.105.127.. 2022-03-01 03:09:41,647 - root - INFO - command executed with nonroot but got an error ['2445061 blocks'] ... 2022-03-01 03:09:41,648 - root - INFO - Waiting 20s for install to complete on 172.23.105.126.. 2022-03-01 03:09:41,895 - root - INFO - command executed with nonroot but got an error ['2445061 blocks'] ... 2022-03-01 03:09:41,895 - root - INFO - Waiting 20s for install to complete on 172.23.105.162.. 2022-03-01 03:09:42,517 - root - INFO - command executed with nonroot but got an error ['2445061 blocks'] ... 2022-03-01 03:09:42,517 - root - INFO - Waiting 20s for install to complete on 172.23.105.158.. 2022-03-01 03:10:01,214 - root - INFO - running command.raw on 172.23.105.127: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:10:01,662 - root - INFO - running command.raw on 172.23.105.126: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:10:01,915 - root - INFO - running command.raw on 172.23.105.162: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:10:02,538 - root - INFO - running command.raw on 172.23.105.158: cd /home/nonroot/; rpm2cpio /home/nonroot/couchbase-server-enterprise-7.1.0-2407-centos7.x86_64.rpm | cpio --extract --make-directories --no-absolute-filenames > /dev/null && echo 1 || echo 0; cd /home/nonroot//opt/couchbase/; ./bin/install/reloc.sh `pwd` > /dev/null && echo 1 || echo 0; 2022-03-01 03:10:21,150 - root - INFO - command executed with nonroot but got an error ['2445061 blocks'] ... 2022-03-01 03:10:21,151 - root - INFO - Waiting 20s for install to complete on 172.23.104.172.. 2022-03-01 03:10:22,056 - root - INFO - command executed with nonroot but got an error ['2445061 blocks'] ... 2022-03-01 03:10:22,057 - root - INFO - Waiting 20s for install to complete on 172.23.106.52.. 2022-03-01 03:10:41,171 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:10:42,077 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:10:42,557 - root - INFO - command executed successfully with nonroot 2022-03-01 03:10:42,558 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.104.172.. 2022-03-01 03:10:43,474 - root - INFO - command executed successfully with nonroot 2022-03-01 03:10:43,474 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.106.52.. 2022-03-01 03:10:47,090 - root - INFO - command executed with nonroot but got an error ['cpio: ./opt/couchbase/LICENSE.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/NOTICES.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/README.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VARIANT.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VERSION.txt not created: newer or same age ve ... 2022-03-01 03:10:47,094 - root - INFO - Waiting 20s for install to complete on 172.23.105.127.. 2022-03-01 03:10:47,471 - root - INFO - command executed with nonroot but got an error ['cpio: ./opt/couchbase/LICENSE.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/NOTICES.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/README.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VARIANT.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VERSION.txt not created: newer or same age ve ... 2022-03-01 03:10:47,474 - root - INFO - Waiting 20s for install to complete on 172.23.105.126.. 2022-03-01 03:10:47,833 - root - INFO - command executed with nonroot but got an error ['cpio: ./opt/couchbase/LICENSE.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/NOTICES.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/README.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VARIANT.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VERSION.txt not created: newer or same age ve ... 2022-03-01 03:10:47,836 - root - INFO - Waiting 20s for install to complete on 172.23.105.162.. 2022-03-01 03:10:48,326 - root - INFO - command executed with nonroot but got an error ['cpio: ./opt/couchbase/LICENSE.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/NOTICES.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/README.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VARIANT.txt not created: newer or same age version exists', 'cpio: ./opt/couchbase/VERSION.txt not created: newer or same age ve ... 2022-03-01 03:10:48,329 - root - INFO - Waiting 20s for install to complete on 172.23.105.158.. 2022-03-01 03:10:52,564 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:10:53,475 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:10:53,975 - root - INFO - command executed successfully with nonroot 2022-03-01 03:10:53,975 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.104.172.. 2022-03-01 03:10:54,882 - root - INFO - command executed successfully with nonroot 2022-03-01 03:10:54,883 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.106.52.. 2022-03-01 03:11:03,985 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:04,893 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:05,346 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:05,346 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.104.172.. 2022-03-01 03:11:06,262 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:06,263 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.106.52.. 2022-03-01 03:11:07,103 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:07,490 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:07,840 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:08,319 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:08,319 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.127.. 2022-03-01 03:11:08,340 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:08,702 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:08,702 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.126.. 2022-03-01 03:11:09,043 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:09,044 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.162.. 2022-03-01 03:11:09,550 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:09,551 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.158.. 2022-03-01 03:11:15,354 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:16,273 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:16,724 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:16,724 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.104.172.. 2022-03-01 03:11:17,666 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:17,667 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.106.52.. 2022-03-01 03:11:18,330 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:18,712 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:19,051 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:19,543 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:19,544 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.127.. 2022-03-01 03:11:19,561 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:19,927 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:19,927 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.126.. 2022-03-01 03:11:20,265 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:20,266 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.162.. 2022-03-01 03:11:20,767 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:20,768 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.158.. 2022-03-01 03:11:26,735 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:27,677 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:28,148 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:28,149 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.104.172.. 2022-03-01 03:11:29,051 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:29,052 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.106.52.. 2022-03-01 03:11:29,552 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:29,929 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:30,276 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:30,754 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:30,755 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.127.. 2022-03-01 03:11:30,775 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:31,128 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:31,128 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.126.. 2022-03-01 03:11:31,497 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:31,497 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.162.. 2022-03-01 03:11:31,986 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:31,986 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.158.. 2022-03-01 03:11:38,159 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:39,062 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:39,538 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:39,539 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.104.172.. 2022-03-01 03:11:40,430 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:40,431 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.106.52.. 2022-03-01 03:11:40,765 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:41,135 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:41,508 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:41,977 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:41,978 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.127.. 2022-03-01 03:11:41,993 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:42,346 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:42,346 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.126.. 2022-03-01 03:11:42,721 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:42,721 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.162.. 2022-03-01 03:11:43,196 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:43,197 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.158.. 2022-03-01 03:11:49,547 - root - INFO - Done with install on 172.23.104.172. 2022-03-01 03:11:49,547 - root - INFO - Waiting for couchbase to be reachable 2022-03-01 03:11:49,558 - root - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/couchbase-cli node-init -c 172.23.104.172 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-03-01 03:11:50,435 - root - INFO - Done with install on 172.23.106.52. 2022-03-01 03:11:50,436 - root - INFO - Waiting for couchbase to be reachable 2022-03-01 03:11:50,445 - root - INFO - running command.raw on 172.23.106.52: /home/nonroot/opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.52 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-03-01 03:11:51,541 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:51,988 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:52,357 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:52,552 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.104.172 2022-03-01 03:11:52,552 - root - INFO - pools/default params : indexMemoryQuota=256 2022-03-01 03:11:52,563 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.104.172 2022-03-01 03:11:52,563 - root - INFO - pools/default params : ftsMemoryQuota=512 2022-03-01 03:11:52,569 - root - INFO - Setting KV memory quota as 1249 MB on 172.23.104.172 2022-03-01 03:11:52,569 - root - INFO - pools/default params : memoryQuota=1249 2022-03-01 03:11:52,574 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:11:52,574 - root - INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:11:52,611 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-01 03:11:52,622 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-01 03:11:52,622 - root - INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password 2022-03-01 03:11:52,731 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:52,764 - root - INFO - --> status:True 2022-03-01 03:11:52,764 - root - INFO - Done with init on 172.23.104.172. 2022-03-01 03:11:52,765 - root - INFO - running command.raw on 172.23.104.172: rm -f *-diag.zip; ls -td /home/nonroot/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-03-01 03:11:52,821 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:52,822 - root - INFO - Done with cleanup on 172.23.104.172. 2022-03-01 03:11:53,142 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:53,193 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:53,193 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.127.. 2022-03-01 03:11:53,201 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:11:53,560 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:53,561 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.126.. 2022-03-01 03:11:53,943 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:53,943 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.162.. 2022-03-01 03:11:54,153 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.106.52 2022-03-01 03:11:54,153 - root - INFO - pools/default params : indexMemoryQuota=256 2022-03-01 03:11:54,164 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.106.52 2022-03-01 03:11:54,164 - root - INFO - pools/default params : ftsMemoryQuota=512 2022-03-01 03:11:54,169 - root - INFO - Setting KV memory quota as 1249 MB on 172.23.106.52 2022-03-01 03:11:54,169 - root - INFO - pools/default params : memoryQuota=1249 2022-03-01 03:11:54,175 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:11:54,175 - root - INFO - /node/controller/setupServices params on 172.23.106.52: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:11:54,215 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-01 03:11:54,224 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-01 03:11:54,225 - root - INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password 2022-03-01 03:11:54,365 - root - INFO - --> status:True 2022-03-01 03:11:54,366 - root - INFO - Done with init on 172.23.106.52. 2022-03-01 03:11:54,366 - root - INFO - running command.raw on 172.23.106.52: rm -f *-diag.zip; ls -td /home/nonroot/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-03-01 03:11:54,399 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:54,399 - root - INFO - Done with cleanup on 172.23.106.52. 2022-03-01 03:11:54,417 - root - INFO - command executed successfully with nonroot 2022-03-01 03:11:54,417 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.158.. 2022-03-01 03:12:03,202 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:12:03,571 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:12:03,953 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:12:04,413 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:04,414 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.127.. 2022-03-01 03:12:04,427 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-server \-- -noinput -detached 2022-03-01 03:12:04,778 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:04,778 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.126.. 2022-03-01 03:12:05,156 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:05,156 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.162.. 2022-03-01 03:12:05,634 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:05,635 - root - INFO - Waiting 10s for couchbase-service to become active on 172.23.105.158.. 2022-03-01 03:12:14,420 - root - INFO - Done with install on 172.23.105.127. 2022-03-01 03:12:14,421 - root - INFO - Waiting for couchbase to be reachable 2022-03-01 03:12:14,428 - root - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.127 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-03-01 03:12:14,780 - root - INFO - Done with install on 172.23.105.126. 2022-03-01 03:12:14,780 - root - INFO - Waiting for couchbase to be reachable 2022-03-01 03:12:14,788 - root - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.126 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-03-01 03:12:15,161 - root - INFO - Done with install on 172.23.105.162. 2022-03-01 03:12:15,162 - root - INFO - Waiting for couchbase to be reachable 2022-03-01 03:12:15,170 - root - INFO - running command.raw on 172.23.105.162: /home/nonroot/opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.162 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-03-01 03:12:15,646 - root - INFO - Done with install on 172.23.105.158. 2022-03-01 03:12:15,646 - root - INFO - Waiting for couchbase to be reachable 2022-03-01 03:12:15,654 - root - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.158 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-03-01 03:12:15,689 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:16,067 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:16,405 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:16,697 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.105.127 2022-03-01 03:12:16,697 - root - INFO - pools/default params : indexMemoryQuota=256 2022-03-01 03:12:16,705 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.105.127 2022-03-01 03:12:16,706 - root - INFO - pools/default params : ftsMemoryQuota=512 2022-03-01 03:12:16,710 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.105.127 2022-03-01 03:12:16,710 - root - INFO - pools/default params : memoryQuota=1379 2022-03-01 03:12:16,713 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:12:16,713 - root - INFO - /node/controller/setupServices params on 172.23.105.127: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:12:16,742 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-01 03:12:16,749 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:16,749 - root - INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:16,812 - root - INFO - --> status:True 2022-03-01 03:12:16,812 - root - INFO - Done with init on 172.23.105.127. 2022-03-01 03:12:16,812 - root - INFO - running command.raw on 172.23.105.127: rm -f *-diag.zip; ls -td /home/nonroot/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-03-01 03:12:16,866 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:16,866 - root - INFO - Done with cleanup on 172.23.105.127. 2022-03-01 03:12:16,928 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:17,075 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.105.126 2022-03-01 03:12:17,075 - root - INFO - pools/default params : indexMemoryQuota=256 2022-03-01 03:12:17,083 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.105.126 2022-03-01 03:12:17,083 - root - INFO - pools/default params : ftsMemoryQuota=512 2022-03-01 03:12:17,087 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.105.126 2022-03-01 03:12:17,087 - root - INFO - pools/default params : memoryQuota=1379 2022-03-01 03:12:17,091 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:12:17,091 - root - INFO - /node/controller/setupServices params on 172.23.105.126: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:12:17,122 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-01 03:12:17,129 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:17,129 - root - INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:17,194 - root - INFO - --> status:True 2022-03-01 03:12:17,194 - root - INFO - Done with init on 172.23.105.126. 2022-03-01 03:12:17,194 - root - INFO - running command.raw on 172.23.105.126: rm -f *-diag.zip; ls -td /home/nonroot/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-03-01 03:12:17,233 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:17,234 - root - INFO - Done with cleanup on 172.23.105.126. 2022-03-01 03:12:17,413 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.105.162 2022-03-01 03:12:17,413 - root - INFO - pools/default params : indexMemoryQuota=256 2022-03-01 03:12:17,420 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.105.162 2022-03-01 03:12:17,420 - root - INFO - pools/default params : ftsMemoryQuota=512 2022-03-01 03:12:17,424 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.105.162 2022-03-01 03:12:17,424 - root - INFO - pools/default params : memoryQuota=1379 2022-03-01 03:12:17,428 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:12:17,428 - root - INFO - /node/controller/setupServices params on 172.23.105.162: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:12:17,459 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-01 03:12:17,465 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:17,465 - root - INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:17,530 - root - INFO - --> status:True 2022-03-01 03:12:17,530 - root - INFO - Done with init on 172.23.105.162. 2022-03-01 03:12:17,530 - root - INFO - running command.raw on 172.23.105.162: rm -f *-diag.zip; ls -td /home/nonroot/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-03-01 03:12:17,592 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:17,592 - root - INFO - Done with cleanup on 172.23.105.162. 2022-03-01 03:12:17,936 - root - INFO - Setting INDEX memory quota as 256 MB on 172.23.105.158 2022-03-01 03:12:17,937 - root - INFO - pools/default params : indexMemoryQuota=256 2022-03-01 03:12:17,943 - root - INFO - Setting FTS memory quota as 512 MB on 172.23.105.158 2022-03-01 03:12:17,943 - root - INFO - pools/default params : ftsMemoryQuota=512 2022-03-01 03:12:17,947 - root - INFO - Setting KV memory quota as 1379 MB on 172.23.105.158 2022-03-01 03:12:17,948 - root - INFO - pools/default params : memoryQuota=1379 2022-03-01 03:12:17,951 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:12:17,951 - root - INFO - /node/controller/setupServices params on 172.23.105.158: 8091:hostname=None&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:12:17,983 - root - INFO - settings/indexes params : storageMode=plasma 2022-03-01 03:12:17,990 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:17,991 - root - INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:18,057 - root - INFO - --> status:True 2022-03-01 03:12:18,057 - root - INFO - Done with init on 172.23.105.158. 2022-03-01 03:12:18,057 - root - INFO - running command.raw on 172.23.105.158: rm -f *-diag.zip; ls -td /home/nonroot/couchbase*.rpm | awk 'NR>2' | xargs rm -f 2022-03-01 03:12:18,102 - root - INFO - command executed successfully with nonroot 2022-03-01 03:12:18,102 - root - INFO - Done with cleanup on 172.23.105.158. 2022-03-01 03:12:18,985 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-03-01 03:12:19,019 - root - INFO - cluster:C1 node:172.23.104.172:8091 version:7.1.0-2407-enterprise aFamily:inet services:['fts', 'index', 'kv', 'n1ql'] 2022-03-01 03:12:19,020 - root - INFO - cluster:C2 node:172.23.106.52:8091 version:7.1.0-2407-enterprise aFamily:inet services:['fts', 'index', 'kv', 'n1ql'] 2022-03-01 03:12:19,020 - root - INFO - cluster:C3 node:172.23.105.158:8091 version:7.1.0-2407-enterprise aFamily:inet services:['fts', 'index', 'kv', 'n1ql'] 2022-03-01 03:12:19,020 - root - INFO - cluster:C4 node:172.23.105.126:8091 version:7.1.0-2407-enterprise aFamily:inet services:['fts', 'index', 'kv', 'n1ql'] 2022-03-01 03:12:19,020 - root - INFO - cluster:C5 node:172.23.105.127:8091 version:7.1.0-2407-enterprise aFamily:inet services:['fts', 'index', 'kv', 'n1ql'] 2022-03-01 03:12:19,020 - root - INFO - cluster:C6 node:172.23.105.162:8091 version:7.1.0-2407-enterprise aFamily:inet services:['fts', 'index', 'kv', 'n1ql'] 2022-03-01 03:12:19,020 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-03-01 03:12:19,020 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-03-01 03:12:19,021 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-03-01 03:12:19,021 - root - INFO - INSTALL COMPLETED ON: 172.23.104.172 2022-03-01 03:12:19,021 - root - INFO - INSTALL COMPLETED ON: 172.23.106.52 2022-03-01 03:12:19,021 - root - INFO - INSTALL COMPLETED ON: 172.23.105.158 2022-03-01 03:12:19,021 - root - INFO - INSTALL COMPLETED ON: 172.23.105.126 2022-03-01 03:12:19,021 - root - INFO - INSTALL COMPLETED ON: 172.23.105.127 2022-03-01 03:12:19,021 - root - INFO - INSTALL COMPLETED ON: 172.23.105.162 2022-03-01 03:12:19,021 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-03-01 03:12:19,021 - root - INFO - TOTAL INSTALL TIME = 257 seconds /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/scripts/install_constants.py:50: DeprecationWarning: invalid escape sequence \ "MACOS_VERSIONS": "/Applications/Couchbase\ Server.app", /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/scripts/install_constants.py:119: DeprecationWarning: invalid escape sequence \C "rm -rf /Applications\Couchbase\ Server.app; " /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/scripts/install_constants.py:140: DeprecationWarning: invalid escape sequence \| "grep 'buildversion.*[Configuration\|Installation] completed successfully.' install_status.txt && echo 1 || echo 0", /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/scripts/install_constants.py:247: DeprecationWarning: invalid escape sequence \- "post_install": NON_ROOT_DOWNLOAD_DIR["LINUX_DISTROS"] + "opt/couchbase/bin/couchbase-server \-- -noinput -detached", success INFO:root:SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 INFO:root:SSH Connected to 172.23.105.158 as nonroot INFO:root:SSH Connected to 172.23.105.126 as nonroot INFO:root:SSH Connected to 172.23.105.127 as nonroot INFO:root:SSH Connected to 172.23.105.162 as nonroot INFO:root:SSH Connected to 172.23.104.172 as nonroot INFO:root:SSH Connected to 172.23.106.52 as nonroot INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.158: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.126: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.162: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.127: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.52: iptables -F INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.104.172: iptables -F INFO:root:command executed with nonroot but got an error ['bash: iptables: command not found'] ... INFO:root:command executed with nonroot but got an error ['bash: iptables: command not found'] ... INFO:root:command executed with nonroot but got an error ['bash: iptables: command not found'] ... INFO:root:command executed with nonroot but got an error ['bash: iptables: command not found'] ... INFO:root:command executed with nonroot but got an error ['bash: iptables: command not found'] ... INFO:root:command executed with nonroot but got an error ['bash: iptables: command not found'] ... 172.23.105.127 bash: iptables: command not found 172.23.105.158 bash: iptables: command not found 172.23.105.126 bash: iptables: command not found 172.23.105.162 bash: iptables: command not found 172.23.106.52 bash: iptables: command not found 172.23.104.172 bash: iptables: command not found Need to set ALLOW_HTP back to True to do git pull branch Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client' Cloning into 'java_sdk_client'... Submodule path 'java_sdk_client': checked out 'b23994ef285cfeb976e9557a630dcf003f95da5e' Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirrors.ocf.berkeley.edu * extras: linux.mirrors.es.net * updates: mirror.hostduplex.com Package maven-3.0.5-17.el7.noarch already installed and latest version Nothing to do find: ‘/root/workspace/*/logs/*’: No such file or directory find: ‘/root/workspace/’: No such file or directory python3: no process found python3 testrunner.py -i /tmp/testexec.30028.ini -c conf/py-rebalance-extended-sanity.conf -p get-cbcollect-info=False,get-logs=False,get-coredumps=False INFO:root:__main__ INFO:__main__:TestRunner: parsing args... INFO:__main__:Checking arguments... INFO:__main__:Conf filename: conf/py-rebalance-extended-sanity.conf INFO:__main__:Test prefix: rebalance.rebalancein.RebalanceInTests INFO:__main__:Test prefix: rebalance.rebalanceinout.RebalanceInOutTests INFO:__main__:Test prefix: rebalance.rebalanceout.RebalanceOutTests INFO:__main__:Test prefix: failover.failovertests.FailoverTests INFO:__main__:Test prefix: failover.AutoFailoverTests.AutoFailoverTests INFO:__main__:TestRunner: start... INFO:__main__:Global Test input params: INFO:__main__: Number of tests initially selected before GROUP filters: 13 INFO:__main__:--> Running test: rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=10000,GROUP=IN;P0 INFO:__main__:Logs folder: /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_1 *** TestRunner *** {'cluster_name': 'testexec.30028', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-coredumps': 'False', 'get-logs': 'False', 'ini': '/tmp/testexec.30028.ini', 'num_nodes': 6, 'spec': 'py-rebalance-extended-sanity'} Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_1 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=10000,GROUP=IN;P0 Test Input params: {'nodes_in': '1', 'nodes_init': '3', 'replicas': '1', 'items': '10000', 'GROUP': 'IN;P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 1, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_1'} Run before suite setup for rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops lib/couchbase_helper/data_analysis_helper.py:905: DeprecationWarning: invalid escape sequence \{ master_key = ".\{12\}$" ./pytests/security/x509main.py:80: DeprecationWarning: invalid escape sequence \. ''' ./pytests/security/x509main.py:90: DeprecationWarning: invalid escape sequence \ def _generate_cert(self, servers, root_cn='Root\ Authority', type='go', encryption="", key_length=1024, client_ip=None, alt_names='default', dns=None, uri=None,wildcard_dns=None): suite_setUp (rebalance.rebalancein.RebalanceInTests) ... -->before_suite_name:rebalance.rebalancein.RebalanceInTests.suite_setUp,suite: ]> 2022-03-01 03:12:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:12:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2407-enterprise 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp============== 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.549217566885412, 'mem_free': 3260780544, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:12:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:12:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:12:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:12:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:12:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:12:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.172, nothing to delete 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.158, nothing to delete 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.126, nothing to delete 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.162, nothing to delete 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 03:12:49 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.172:8091 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.172:8091 is running 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.52:8091 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.52:8091 is running 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.158:8091 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.158:8091 is running 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.126:8091 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.126:8091 is running 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.127:8091 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.127:8091 is running 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.162:8091 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.162:8091 is running 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-03-01 03:12:49 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '124', 'memoryTotal': 3947372544, 'memoryFree': 3260416000, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.104.172 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.104.172 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1249 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:12:50 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:12:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '124', 'memoryTotal': 3947372544, 'memoryFree': 3255746560, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2017 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:12:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '101', 'memoryTotal': 4201627648, 'memoryFree': 3565387776, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:12:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '101', 'memoryTotal': 4201627648, 'memoryFree': 3557699584, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:12:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '101', 'memoryTotal': 4201627648, 'memoryFree': 3550654464, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '101', 'memoryTotal': 4201627648, 'memoryFree': 3549163520, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1379, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:12:55 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:12:55 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** 2022-03-01 03:12:55 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 03:12:55 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."' 2022-03-01 03:12:56 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ... 2022-03-01 03:13:01 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-03-01 03:13:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster 2022-03-01 03:13:01 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.52:8091 to cluster 2022-03-01 03:13:01 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 2022-03-01 03:13:11 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 03:13:11 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:13:30 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.158:8091 to cluster 2022-03-01 03:13:30 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 2022-03-01 03:13:40 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-03-01 03:13:40 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:13:55 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.105.158 not part of cluster inactiveAdded 2022-03-01 03:13:55 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.52 not part of cluster inactiveAdded 2022-03-01 03:13:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-01 03:13:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 03:14:05 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 03:14:05 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.01826000213623 sec 2022-03-01 03:14:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:14:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:14:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:14:06 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:14:06 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.172 2022-03-01 03:14:06 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 03:14:06 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default 2022-03-01 03:14:06 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.172 to accept set ops 2022-03-01 03:14:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:14:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:14:09 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1249 2022-03-01 03:14:09 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ============== 2022-03-01 03:14:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:14:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:14:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:14:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:14:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:14:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:14:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 10.01013684744045, 'mem_free': 3150880768, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.242070540471961, 'mem_free': 3489976320, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 18.44961240310078, 'mem_free': 3174227968, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 114.66684341430664 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:26 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default ./lib/log_scanner.py:14: DeprecationWarning: invalid escape sequence \s "Basic\s[a-zA-Z]\{10,\}==", ./lib/log_scanner.py:15: DeprecationWarning: invalid escape sequence \[ "Menelaus-Auth-User:\[", 2022-03-01 03:14:27 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:27 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:27 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:27 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:14:27 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:27 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:27 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:27 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:14:27 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:14:27 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:14:27 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:14:27 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:14:28 | INFO | MainProcess | MainThread | [basetestcase.check_snap_start_corruption] Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:14:28 | INFO | MainProcess | MainThread | [basetestcase._record_vbuckets] Record vbucket for the bucket default 2022-03-01 03:14:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:14:28 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:14:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:14:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:14:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:14:30 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:14:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:14:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:31 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 10.01013684744045, 'mem_free': 3150880768, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.219419924337957, 'mem_free': 3493330944, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.691492047462762, 'mem_free': 3177672704, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:14:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:33 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:14:33 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:14:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:14:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:14:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:14:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:14:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:14:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ============== 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.104.172 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 172.23.104.172 existing buckets : [] 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.104.172 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.158, nothing to delete 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.126, nothing to delete 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.162, nothing to delete 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 03:14:50 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-03-01 03:14:50 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} 2022-03-01 03:14:51 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started 2022-03-01 03:14:51 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-03-01 03:15:01 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2022-03-01 03:15:11 | INFO | MainProcess | MainThread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2022-03-01 03:15:31 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] rebalance progress took 40.06 seconds 2022-03-01 03:15:31 | INFO | MainProcess | MainThread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.172:8091 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.172:8091 is running 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.52:8091 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.52:8091 is running 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.158:8091 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.158:8091 is running 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.126:8091 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.126:8091 is running 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.127:8091 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.127:8091 is running 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.162:8091 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.162:8091 is running 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ============== 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 03:15:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections ok ---------------------------------------------------------------------- Ran 1 test in 189.659s OK rebalance_in_after_ops (rebalance.rebalancein.RebalanceInTests) ... Cluster instance shutdown with force -->result: 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.1.0-2407-enterprise 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 rebalance_in_after_ops============== 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 0.9831106629694983, 'mem_free': 3166781440, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} 2022-03-01 03:15:41 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 03:15:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:15:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:15:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:15:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:15:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:15:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:15:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:15:46 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 rebalance_in_after_ops ============== 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.104.172, nothing to delete 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.158, nothing to delete 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.126, nothing to delete 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.162, nothing to delete 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 03:15:58 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.172:8091 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.172:8091 is running 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.52:8091 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.52:8091 is running 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.158:8091 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:58 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.158:8091 is running 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.126:8091 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.126:8091 is running 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.127:8091 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.127:8091 is running 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.162:8091 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.162:8091 is running 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 rebalance_in_after_ops ============== 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force 2022-03-01 03:15:59 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '312', 'memoryTotal': 3947372544, 'memoryFree': 3169751040, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.104.172 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.104.172 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1249 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts 2022-03-01 03:16:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:16:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '314', 'memoryTotal': 3947372544, 'memoryFree': 3309146112, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} 2022-03-01 03:16:01 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2017 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:16:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '290', 'memoryTotal': 4201627648, 'memoryFree': 3621965824, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:16:02 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:16:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '292', 'memoryTotal': 4201627648, 'memoryFree': 3540398080, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '292', 'memoryTotal': 4201627648, 'memoryFree': 3531784192, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:16:03 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self 2022-03-01 03:16:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '292', 'memoryTotal': 4201627648, 'memoryFree': 3530338304, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,password,8091 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:16:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). 2022-03-01 03:16:06 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2022-03-01 03:16:06 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** 2022-03-01 03:16:06 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2022-03-01 03:16:11 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2022-03-01 03:16:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2022-03-01 03:16:12 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.52:8091 to cluster 2022-03-01 03:16:12 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 2022-03-01 03:16:22 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 03:16:22 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:16:38 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.158:8091 to cluster 2022-03-01 03:16:38 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 2022-03-01 03:16:48 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.02 seconds 2022-03-01 03:16:48 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:17:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.105.158 not part of cluster inactiveAdded 2022-03-01 03:17:03 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.106.52 not part of cluster inactiveAdded 2022-03-01 03:17:03 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-01 03:17:03 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 03:17:13 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 03:17:13 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.023096561431885 sec 2022-03-01 03:17:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:13 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:17:13 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2022-03-01 03:17:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:14 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] ['ok'] 2022-03-01 03:17:14 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.104.172 2022-03-01 03:17:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore 2022-03-01 03:17:14 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default 2022-03-01 03:17:14 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.104.172 to accept set ops 2022-03-01 03:17:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:15 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:16 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 1249 2022-03-01 03:17:16 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 rebalance_in_after_ops ============== 2022-03-01 03:17:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:16 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:17:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:17:17 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:17:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:17:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:17:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.937452519625221, 'mem_free': 3099168768, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.318548387096774, 'mem_free': 3466043392, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.1025641025641, 'mem_free': 3148791808, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 112.24471855163574 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:33 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:34 | INFO | Process-7 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:34 | INFO | Process-9 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:34 | INFO | Process-10 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:34 | INFO | Process-8 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:34 | INFO | Process-7 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:34 | INFO | Process-10 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:34 | INFO | Process-9 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:34 | INFO | Process-8 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:34 | INFO | Process-7 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:34 | INFO | Process-10 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:34 | INFO | Process-9 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:34 | INFO | Process-8 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:35 | INFO | MainProcess | test_thread | [basetestcase.check_snap_start_corruption] Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:17:35 | INFO | MainProcess | test_thread | [basetestcase._record_vbuckets] Record vbucket for the bucket default 2022-03-01 03:17:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:17:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:17:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:17:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:17:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:17:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:17:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:39 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:17:39 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:17:39 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:17:52 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:17:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:17:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:17:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:17:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:17:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:17:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:17:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:17:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:17:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:17:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:17:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:06 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket 2022-03-01 03:18:06 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:18:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:18:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:10 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket 2022-03-01 03:18:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. ... 2022-03-01 03:18:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:30 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:18:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:18:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:18:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:18:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:18:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:18:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:18:39 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:18:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:18:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:18:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:18:43 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:18:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:18:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:44 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:18:46 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:18:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:47 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:18:49 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:18:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:50 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:18:50 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:18:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:18:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:51 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:18:53 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:18:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:18:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:54 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:18:55 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:18:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:18:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:18:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:18:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:18:56 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:18:57 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:18:57 | INFO | MainProcess | test_thread | [basetestcase.get_and_compare_active_replica_data_set_all] Begin Verification for Active Vs Replica 2022-03-01 03:18:57 | INFO | MainProcess | test_thread | [basetestcase.get_and_compare_active_replica_data_set_all] End Verification for Active Vs Replica 2022-03-01 03:18:58 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.126:8091 to cluster 2022-03-01 03:18:58 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 2022-03-01 03:19:08 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] rebalance progress took 10.01 seconds 2022-03-01 03:19:08 | INFO | MainProcess | Cluster_Thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:19:22 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes] Node 172.23.105.126 not part of cluster inactiveAdded 2022-03-01 03:19:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} 2022-03-01 03:19:23 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2022-03-01 03:19:33 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-03-01 03:19:33 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2022-03-01 03:19:53 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2022-03-01 03:19:53 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 30.063600540161133 sec 2022-03-01 03:19:53 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:19:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:19:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:19:54 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:19:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:19:54 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:19:54 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:19:55 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:19:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:19:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:19:55 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:19:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:19:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:19:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:19:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:19:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:19:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:19:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:19:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:19:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:19:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:19:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:19:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:06 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:10 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:10 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:20:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:20:10 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:20:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:20:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:11 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:11 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:11 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:12 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-03-01 03:20:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:14 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket 2022-03-01 03:20:14 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:14 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-03-01 03:20:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:18 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket 2022-03-01 03:20:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:18 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:18 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:19 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-03-01 03:20:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:20 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket 2022-03-01 03:20:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp 2022-03-01 03:20:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:23 | INFO | MainProcess | Cluster_Thread | [task.check] Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket 2022-03-01 03:20:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.104.172:11210 default 2022-03-01 03:20:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.158:11210 default 2022-03-01 03:20:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.126:11210 default 2022-03-01 03:20:23 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.52:11210 default 2022-03-01 03:20:23 | INFO | MainProcess | test_thread | [task.__init__] 10000 items will be verified on default bucket 2022-03-01 03:20:24 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified 2022-03-01 03:20:26 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified 2022-03-01 03:20:26 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified in 2.921344757080078 sec.the average number of ops - 3423.0 per second 2022-03-01 03:20:26 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:20:26 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:20:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:27 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:27 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:28 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:28 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:28 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:31 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:32 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:32 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:32 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:32 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:32 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:33 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:33 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:33 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:33 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:33 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:33 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:34 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:34 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:34 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:34 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:34 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:35 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:36 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:36 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:36 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:36 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:37 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:37 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:37 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:38 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:38 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:38 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all 2022-03-01 03:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:43 | INFO | MainProcess | Cluster_Thread | [task.check] Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket 2022-03-01 03:20:43 | INFO | MainProcess | test_thread | [rebalance_helper.wait_till_total_numbers_match] waiting for sum_of_curr_items == total_items.... 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] curr_items from 172.23.104.172:8091 : 2510 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_pending_num from 172.23.104.172:8091 : 0 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_active_num from 172.23.104.172:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_replica_num from 172.23.104.172:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] curr_items from 172.23.105.126:8091 : 2486 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_pending_num from 172.23.105.126:8091 : 0 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_active_num from 172.23.105.126:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_replica_num from 172.23.105.126:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] curr_items from 172.23.105.158:8091 : 2490 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_pending_num from 172.23.105.158:8091 : 0 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_active_num from 172.23.105.158:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_replica_num from 172.23.105.158:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] curr_items from 172.23.106.52:8091 : 2514 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_pending_num from 172.23.106.52:8091 : 0 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_active_num from 172.23.106.52:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] vb_replica_num from 172.23.106.52:8091 : 256 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 2022-03-01 03:20:44 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] sum : 10000 and sum * (replica_factor + 1) (2) : 20000 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] curr_items_tot from master: 20000 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [rebalance_helper.verify_items_count] delta : 0 missing_percentage : 1 replica_factor : 1 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [basetestcase.compare_failovers_logs] Begin Verification for failovers logs comparison 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:20:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:20:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:20:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [basetestcase.compare_failovers_logs] End Verification for Failovers logs comparison 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [basetestcase.compare_vbucket_seqnos] Begin Verification for vbucket sequence numbers comparison 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:20:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:20:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:20:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:20:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:20:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:20:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:20:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:20:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:20:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:20:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:20:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:21:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:00 | INFO | MainProcess | test_thread | [basetestcase.compare_vbucket_seqnos] End Verification for vbucket sequence numbers comparison 2022-03-01 03:21:00 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ... 2022-03-01 03:21:30 | INFO | MainProcess | test_thread | [basetestcase.data_analysis_active_replica_all] Begin Verification for data comparison 2022-03-01 03:21:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:30 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:21:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:31 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:32 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:21:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:33 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:35 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:35 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:21:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:36 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:36 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:36 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:37 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:37 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:38 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:38 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:21:38 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:39 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:39 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:40 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:21:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:41 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:43 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:21:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:44 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:44 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:21:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:45 | INFO | MainProcess | test_thread | [remote_util.get_data_map_using_cbtransfer] Run cbtransfer to get data map 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [remote_util.execute_cbtransfer] done execute cbtransfer 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [basetestcase.data_analysis_active_replica_all] Comparing :: Prev vs Current :: Active and Replica 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [basetestcase.data_analysis_active_replica_all] Comparing :: Current :: Active and Replica 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [basetestcase.data_analysis_active_replica_all] End Verification for data comparison 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [basetestcase.vb_distribution_analysis] Begin Verification for vb_distribution_analysis 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:46 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:47 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:21:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:21:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [basetestcase.vb_distribution_analysis] Begin Verification for Bucket default 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [basetestcase.vb_distribution_analysis] End Verification for vb_distribution_analysis 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [basetestcase.check_snap_start_corruption] Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [basetestcase._record_vbuckets] Record vbucket for the bucket default 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:21:51 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:21:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:21:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:21:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:21:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:21:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:21:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with nonroot 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.583756345177665, 'mem_free': 3031093248, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.126:8091 => {'services': ['kv'], 'cpu_utilization': 3.371069182389937, 'mem_free': 3506843648, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.219364599092285, 'mem_free': 3381604352, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.210392902408111, 'mem_free': 3052879872, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.104.172 as nonroot 2022-03-01 03:21:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.106.52 as nonroot 2022-03-01 03:21:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.158 as nonroot 2022-03-01 03:21:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.126 as nonroot 2022-03-01 03:21:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:21:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:21:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 2022-03-01 03:21:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.127 as nonroot 2022-03-01 03:21:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:22:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:22:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 2022-03-01 03:22:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 172.23.105.162 as nonroot 2022-03-01 03:22:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-03-01 03:22:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-03-01 03:22:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 rebalance_in_after_ops ============== 2022-03-01 03:22:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 172.23.104.172 2022-03-01 03:22:14 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.104.172 existing buckets : [] 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.104.172 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.106.52, nothing to delete 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.158, nothing to delete 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.126, nothing to delete 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.127, nothing to delete 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 172.23.105.162, nothing to delete 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'... 2022-03-01 03:22:15 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."' 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2022-03-01 03:22:15 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2022-03-01 03:22:25 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 62.00 % 2022-03-01 03:22:35 | INFO | MainProcess | test_thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 62.00 % 2022-03-01 03:22:55 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 40.05 seconds 2022-03-01 03:22:55 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.104.172:8091 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.104.172:8091 is running 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.52:8091 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.52:8091 is running 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.158:8091 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.158:8091 is running 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.126:8091 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.126:8091 is running 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.127:8091 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.127:8091 is running 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.162:8091 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running? 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.162:8091 is running 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 rebalance_in_after_ops ============== 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections 2022-03-01 03:23:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_1 ok ---------------------------------------------------------------------- Ran 1 test in 444.102s OK rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_2 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=10000,doc_ops=create:update:delete,value_size=1024,GROUP=IN;P0 Test Input params: {'nodes_in': '3', 'replicas': '1', 'items': '10000', 'doc_ops': 'create:update:delete', 'value_size': '1024', 'GROUP': 'IN;P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 2, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_2'} [2022-03-01 03:23:05,290] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:05,428] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:23:05,706] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:06,108] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:06,117] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:23:06,132] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:23:06,132] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #2 rebalance_in_with_ops============== [2022-03-01 03:23:06,132] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:23:06,296] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:23:06,302] - [task:159] INFO - {'uptime': '740', 'memoryTotal': 3947372544, 'memoryFree': 3072237568, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:23:06,310] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:23:06,310] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:23:06,310] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:23:06,321] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:23:06,321] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:23:06,321] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:23:06,332] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:23:06,343] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:23:06,343] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:23:06,346] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:23:06,347] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:23:06,347] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:23:06,347] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:23:06,471] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:23:06,473] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:06,608] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:23:06,889] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:07,309] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:07,310] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:07,390] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:07,390] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:07,393] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:07,396] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:07,403] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:23:07,430] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:23:07,434] - [task:159] INFO - {'uptime': '738', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 3764, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:23:07,436] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:23:07,436] - [rest_client:1248] INFO - pools/default params : memoryQuota=2521 [2022-03-01 03:23:07,442] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:23:07,442] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:23:07,572] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:23:07,573] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:07,763] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:23:08,149] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:08,715] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:08,716] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:08,874] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:08,874] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:08,877] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:08,881] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:08,887] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:23:08,954] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:23:09,004] - [task:159] INFO - {'uptime': '714', 'memoryTotal': 4201627648, 'memoryFree': 3593093120, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:23:09,053] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:23:09,053] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:23:09,103] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:23:09,104] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:23:09,208] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:23:09,210] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:09,469] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:23:09,799] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:10,123] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:10,124] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:10,192] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:10,192] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:10,195] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:10,197] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:10,201] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:23:10,214] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:23:10,216] - [task:159] INFO - {'uptime': '719', 'memoryTotal': 4201627648, 'memoryFree': 3614793728, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:23:10,218] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:23:10,219] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:23:10,223] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:23:10,223] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:23:10,282] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:23:10,284] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:10,391] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:23:10,700] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:11,036] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:11,036] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:11,106] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:11,106] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:11,109] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:11,111] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:11,115] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:23:11,127] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:23:11,130] - [task:159] INFO - {'uptime': '719', 'memoryTotal': 4201627648, 'memoryFree': 3534950400, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:23:11,134] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:23:11,137] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:23:11,138] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:23:11,188] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:23:11,189] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:11,293] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:23:11,577] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:11,916] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:11,916] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:11,985] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:11,985] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:11,988] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:11,991] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:11,995] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:23:12,008] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:23:12,011] - [task:159] INFO - {'uptime': '719', 'memoryTotal': 4201627648, 'memoryFree': 3529605120, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:23:12,015] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:23:12,018] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:23:12,018] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:23:12,069] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:23:12,071] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:12,172] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:23:12,487] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:12,816] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:12,817] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:12,886] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:12,887] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:12,890] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:12,892] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:23:12,896] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:23:12,907] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:23:13,071] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:23:18,076] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:23:18,094] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:23:18,096] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:18,233] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:23:18,524] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:18,906] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:18,907] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:23:18,984] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:18,985] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:23:18,985] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 03:23:19,935] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 03:23:19,966] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 03:23:19,966] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 03:23:20,583] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:20,691] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:22,450] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 03:23:22,452] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #2 rebalance_in_with_ops ============== [2022-03-01 03:23:22,463] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:22,606] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:23:22,901] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:23,295] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:23,298] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:23,440] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:23:23,711] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:24,101] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:24,104] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:24,213] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:23:24,564] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:24,939] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:24,942] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:25,055] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:23:25,407] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:25,780] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:25,783] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:25,933] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:23:26,254] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:26,617] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:26,620] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:26,769] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:23:27,096] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:27,467] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:39,997] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:23:39,997] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 18.68843960601348, 'mem_free': 2991292416, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:23:39,997] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:23:39,997] - [basetestcase:559] INFO - Time to execute basesetup : 34.70905327796936 [2022-03-01 03:23:40,066] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:41,063] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:41,063] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:41,064] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:41,065] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:42,572] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 03:23:42,573] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:23:42,641] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:23:42,781] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:23:43,083] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:23:43,498] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:23:43,499] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:23:44,398] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:23:44,470] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:44,601] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:44,781] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:23:44,999] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:23:45,012] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:23:55,029] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:23:55,029] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:24:11,925] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 03:24:11,941] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:24:21,957] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:24:21,957] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:24:36,005] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 03:24:36,022] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:24:46,039] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:24:46,039] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:25:00,648] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 03:25:00,648] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 03:25:00,649] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 03:25:00,682] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:25:00,754] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:25:10,769] - [rest_client:2012] INFO - rebalance percentage : 27.00 % [2022-03-01 03:25:10,769] - [task:843] INFO - Rebalance - status: running, progress: 27.00% [2022-03-01 03:25:30,810] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:25:30,821] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.067564487457275 sec [2022-03-01 03:25:30,822] - [basetestcase:829] INFO - sleep for 10 secs. Sleep before stat verification ... [2022-03-01 03:25:40,852] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:25:40,862] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:25:40,872] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:25:40,882] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:25:41,838] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:41,977] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:25:42,252] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:42,632] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:42,633] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:42,927] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:42,929] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:25:42,931] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:43,068] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:25:43,351] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:43,740] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:43,741] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:44,034] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:44,036] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:25:44,038] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:44,146] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:25:44,470] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:44,808] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:44,809] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:44,984] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:44,985] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:25:44,987] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:45,092] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:25:45,398] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:45,736] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:45,737] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:45,910] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:45,912] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:25:46,019] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:25:46,101] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:25:46,165] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:25:46,265] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:25:46,400] - [task:2319] INFO - 14999 items will be verified on default bucket [2022-03-01 03:25:46,914] - [task:2328] INFO - 0 items were verified [2022-03-01 03:25:52,015] - [task:2328] INFO - 10000 items were verified [2022-03-01 03:25:54,366] - [task:2332] INFO - 14999 items were verified in 7.965174198150635 sec.the average number of ops - 1883.0 per second [2022-03-01 03:25:54,385] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:25:54,394] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:25:54,960] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:55,098] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:25:55,377] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:55,771] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:55,772] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:56,033] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:56,037] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:56,177] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:25:56,445] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:56,828] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:56,829] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:57,123] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:57,126] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:57,227] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:25:57,537] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:57,876] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:57,877] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:58,049] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:58,053] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:58,156] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:25:58,456] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:58,786] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:58,787] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:25:58,958] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:25:58,960] - [task:972] INFO - Saw curr_items 9999 == 9999 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket [2022-03-01 03:25:58,962] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:25:59,097] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:25:59,366] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:25:59,751] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:25:59,752] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:00,014] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:00,017] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:00,159] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:26:00,429] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:00,805] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:00,806] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:01,088] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:01,092] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:01,197] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:26:01,501] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:01,833] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:01,834] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:02,002] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:02,005] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:02,106] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:26:02,413] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:02,750] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:02,751] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:02,931] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:02,933] - [task:972] INFO - Saw vb_active_curr_items 9999 == 9999 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket [2022-03-01 03:26:02,935] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:03,072] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:26:03,338] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:03,730] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:03,730] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:03,988] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:03,991] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:04,125] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:26:04,388] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:04,771] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:04,772] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:05,080] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:05,083] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:05,187] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:26:05,494] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:05,826] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:05,827] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:06,000] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:06,003] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:06,106] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:26:06,409] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:06,747] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:06,747] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:06,920] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:06,921] - [task:972] INFO - Saw vb_replica_curr_items 9999 == 9999 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket [2022-03-01 03:26:06,923] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:07,060] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:26:07,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:07,706] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:07,707] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:08,003] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:08,006] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:08,149] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:26:08,418] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:08,812] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:08,813] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:09,074] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:09,077] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:09,181] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:26:09,492] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:09,827] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:09,828] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:10,001] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:10,004] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:10,106] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:26:10,409] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:10,741] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:10,742] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:26:10,876] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:10,877] - [task:972] INFO - Saw curr_items_tot 19998 == 19998 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091',default bucket [2022-03-01 03:26:10,877] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:26:11,997] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 2495 [2022-03-01 03:26:11,997] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:26:11,997] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 256 [2022-03-01 03:26:11,997] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 256 [2022-03-01 03:26:11,997] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 2510 [2022-03-01 03:26:11,997] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 03:26:11,997] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 256 [2022-03-01 03:26:11,997] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 256 [2022-03-01 03:26:11,997] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 2499 [2022-03-01 03:26:11,998] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:26:11,998] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 256 [2022-03-01 03:26:11,998] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 256 [2022-03-01 03:26:11,998] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 2495 [2022-03-01 03:26:11,998] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:26:11,998] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 256 [2022-03-01 03:26:11,998] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 256 [2022-03-01 03:26:11,998] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:26:11,998] - [rebalance_helper:390] INFO - sum : 9999 and sum * (replica_factor + 1) (2) : 19998 [2022-03-01 03:26:12,380] - [rebalance_helper:393] INFO - curr_items_tot from master: 19998 [2022-03-01 03:26:12,380] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:26:12,400] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:26:12,400] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:26:12,532] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:12,666] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:26:12,941] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:13,322] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:13,323] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:26:13,906] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:13,916] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:14,018] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:26:14,321] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:14,656] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:14,657] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:26:14,950] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:14,960] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:15,065] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:26:15,370] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:15,707] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:15,707] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:26:16,006] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:16,017] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:16,152] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:26:16,424] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:16,800] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:16,801] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:26:17,383] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:26:17,440] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:26:17,441] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.77361357305647, 'mem_free': 2947592192, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:26:17,441] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['kv'], 'cpu_utilization': 3.629032258064516, 'mem_free': 3506507776, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:26:17,441] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['kv'], 'cpu_utilization': 4.832620186257237, 'mem_free': 3480739840, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:26:17,441] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['kv'], 'cpu_utilization': 7.649583438525625, 'mem_free': 3172245504, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:26:17,441] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:26:17,443] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:17,577] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:26:17,839] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:18,221] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:18,224] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:18,359] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:26:18,620] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:19,008] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:19,012] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:19,113] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:26:19,418] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:19,752] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:19,754] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:19,856] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:26:20,156] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:20,495] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:20,498] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:20,601] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:26:20,896] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:21,235] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:21,238] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:26:21,342] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:26:21,650] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:26:21,982] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:26:34,696] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #2 rebalance_in_with_ops ============== [2022-03-01 03:26:34,747] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 03:26:35,514] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 03:26:35,519] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 03:26:35,520] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 03:26:35,530] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 03:26:35,537] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 03:26:35,544] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 03:26:35,549] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 03:26:35,555] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 03:26:35,556] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 03:26:35,578] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 03:26:35,578] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 03:26:35,585] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:26:35,604] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 03:26:35,606] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:26:35,662] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:26:35,665] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:26:55,693] - [rest_client:1951] INFO - rebalance progress took 20.03 seconds [2022-03-01 03:26:55,694] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:27:05,726] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 03:27:05,734] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 03:27:05,734] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:05,741] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 03:27:05,745] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 03:27:05,745] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:05,749] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 03:27:05,752] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 03:27:05,752] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:05,755] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 03:27:05,758] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 03:27:05,758] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:05,761] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 03:27:05,765] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 03:27:05,766] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:05,769] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 03:27:05,773] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 03:27:05,773] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:05,776] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 03:27:05,776] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #2 rebalance_in_with_ops ============== [2022-03-01 03:27:05,777] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 03:27:05,777] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_2 ok ---------------------------------------------------------------------- Ran 1 test in 240.498s OK incremental_rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_3 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalancein.RebalanceInTests.incremental_rebalance_in_with_ops,replicas=1,items=10000,doc_ops=update,GROUP=IN;P0 Test Input params: {'replicas': '1', 'items': '10000', 'doc_ops': 'update', 'GROUP': 'IN;P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 3, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_3'} [2022-03-01 03:27:05,802] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:05,939] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:27:06,236] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:06,621] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:06,629] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:27:06,645] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:27:06,646] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #3 incremental_rebalance_in_with_ops============== [2022-03-01 03:27:06,646] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:27:06,808] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:27:06,813] - [task:159] INFO - {'uptime': '981', 'memoryTotal': 3947372544, 'memoryFree': 2983575552, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:27:06,821] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:27:06,821] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:27:06,821] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:27:06,831] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:27:06,831] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:27:06,831] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:27:06,844] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:27:06,854] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:27:06,854] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:27:06,856] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:27:06,857] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:27:06,857] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:27:06,857] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:27:06,982] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:27:06,984] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:07,126] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:27:07,400] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:07,776] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:07,777] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:07,815] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:07,815] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:07,818] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:07,821] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:07,828] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:27:07,858] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:27:07,861] - [task:159] INFO - {'uptime': '978', 'memoryTotal': 0, 'memoryFree': 0, 'mcdMemoryReserved': 3764, 'mcdMemoryAllocated': 0, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:27:07,864] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:27:07,864] - [rest_client:1248] INFO - pools/default params : memoryQuota=2521 [2022-03-01 03:27:07,869] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:27:07,869] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:27:08,000] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:27:08,002] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:08,141] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:27:08,412] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:08,797] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:08,798] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:08,838] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:08,838] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:08,842] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:08,845] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:08,851] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:27:08,870] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:27:08,874] - [task:159] INFO - {'uptime': '954', 'memoryTotal': 4201627648, 'memoryFree': 3597336576, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:27:08,876] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:27:08,876] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:27:08,879] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:27:08,880] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:27:08,938] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:27:08,940] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:09,047] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:27:09,360] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:09,689] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:09,689] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:09,758] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:09,758] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:09,762] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:09,764] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:09,768] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:27:09,781] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:27:09,785] - [task:159] INFO - {'uptime': '959', 'memoryTotal': 4201627648, 'memoryFree': 3610943488, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:27:09,787] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:27:09,787] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:27:09,790] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:27:09,791] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:27:09,854] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:27:09,856] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:09,962] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:27:10,274] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:10,619] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:10,619] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:10,689] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:10,690] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:10,693] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:10,695] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:10,698] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:27:10,711] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:27:10,714] - [task:159] INFO - {'uptime': '961', 'memoryTotal': 4201627648, 'memoryFree': 3531796480, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:27:10,717] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:27:10,721] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:27:10,721] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:27:10,777] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:27:10,779] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:10,888] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:27:11,174] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:11,513] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:11,514] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:11,584] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:11,584] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:11,587] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:11,590] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:11,594] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:27:11,609] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:27:11,612] - [task:159] INFO - {'uptime': '961', 'memoryTotal': 4201627648, 'memoryFree': 3528396800, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:27:11,615] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:27:11,619] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:27:11,619] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:27:11,670] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:27:11,672] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:11,776] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:27:12,085] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:12,422] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:12,423] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:12,492] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:12,492] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:12,495] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:12,498] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:27:12,503] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:27:12,514] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:27:12,682] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:27:17,687] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:27:17,707] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:27:17,709] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:17,847] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:27:18,113] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:18,493] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:18,494] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:27:18,533] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:18,533] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:27:18,533] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 03:27:19,545] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 03:27:19,577] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 03:27:19,577] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 03:27:20,194] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:20,322] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:20,865] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 03:27:20,867] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #3 incremental_rebalance_in_with_ops ============== [2022-03-01 03:27:20,878] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:21,017] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:27:21,296] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:21,682] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:21,685] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:21,823] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:27:22,109] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:22,505] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:22,508] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:22,610] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:27:22,913] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:23,246] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:23,250] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:23,353] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:27:23,662] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:24,006] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:24,009] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:24,111] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:27:24,390] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:24,728] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:24,731] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:24,837] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:27:25,150] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:25,482] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:37,672] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:27:37,672] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.21254801536492, 'mem_free': 2911928320, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:27:37,672] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:27:37,672] - [basetestcase:559] INFO - Time to execute basesetup : 31.872114181518555 [2022-03-01 03:27:37,720] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:38,741] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:38,742] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:38,743] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:38,744] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:39,517] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 03:27:39,518] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:27:39,545] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:27:39,693] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:27:39,968] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:27:40,352] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:27:40,352] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:27:41,228] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:27:41,305] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:27:41,379] - [basetestcase:829] INFO - sleep for 30 secs. ... [2022-03-01 03:27:41,674] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:27:41,686] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:27:51,704] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:27:51,704] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:28:08,482] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 03:28:08,496] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:28:18,513] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:28:18,513] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:28:32,235] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 03:28:32,235] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 03:28:32,270] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:28:32,327] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:28:34,910] - [data_helper:789] INFO - Received forward map, reset vbucket map, new direct_client [2022-03-01 03:28:34,920] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:28:35,367] - [data_helper:789] INFO - Received forward map, reset vbucket map, new direct_client [2022-03-01 03:28:35,384] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:28:42,350] - [rest_client:2012] INFO - rebalance percentage : 28.00 % [2022-03-01 03:28:42,350] - [task:843] INFO - Rebalance - status: running, progress: 28.00% [2022-03-01 03:29:02,398] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:29:02,409] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.08105778694153 sec [2022-03-01 03:29:02,419] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:29:02,429] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:29:02,439] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:29:02,448] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:29:03,412] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:03,549] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:29:03,828] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:04,209] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:04,210] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:04,502] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:04,504] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:29:04,506] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:04,648] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:29:04,929] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:05,310] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:05,311] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:05,563] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:05,564] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:29:05,566] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:05,671] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:29:05,991] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:06,327] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:06,328] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:06,501] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:06,502] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:29:06,570] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:29:06,648] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:29:06,746] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:29:06,866] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 03:29:07,504] - [task:2328] INFO - 0 items were verified [2022-03-01 03:29:09,654] - [task:2328] INFO - 10000 items were verified [2022-03-01 03:29:09,654] - [task:2332] INFO - 10000 items were verified in 2.78761625289917 sec.the average number of ops - 3587.0 per second [2022-03-01 03:29:09,673] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:29:09,683] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:29:10,531] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:10,666] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:29:10,937] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:11,324] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:11,325] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:11,620] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:11,623] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:11,761] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:29:12,044] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:12,427] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:12,427] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:12,719] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:12,722] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:12,824] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:29:13,138] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:13,472] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:13,473] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:13,643] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:13,644] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:29:13,647] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:13,780] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:29:14,054] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:14,430] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:14,431] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:14,725] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:14,728] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:14,863] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:29:15,136] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:15,517] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:15,518] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:15,812] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:15,815] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:15,916] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:29:16,226] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:16,561] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:16,562] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:16,733] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:16,734] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:29:16,736] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:16,879] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:29:17,158] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:17,540] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:17,541] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:17,833] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:17,837] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:17,980] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:29:18,250] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:18,631] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:18,632] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:18,927] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:18,930] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:19,035] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:29:19,356] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:19,695] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:19,696] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:19,869] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:19,871] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:29:19,873] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:20,005] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:29:20,270] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:20,654] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:20,654] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:20,911] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:20,915] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:21,050] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:29:21,320] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:21,698] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:21,699] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:22,005] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:22,009] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:22,114] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:29:22,429] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:22,763] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:22,764] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:29:22,942] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:22,944] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:29:22,944] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:29:23,831] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 3318 [2022-03-01 03:29:23,832] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:29:23,832] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 342 [2022-03-01 03:29:23,832] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 342 [2022-03-01 03:29:23,832] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 3340 [2022-03-01 03:29:23,832] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:29:23,832] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 341 [2022-03-01 03:29:23,832] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 341 [2022-03-01 03:29:23,832] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 3342 [2022-03-01 03:29:23,833] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:29:23,833] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 341 [2022-03-01 03:29:23,833] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 341 [2022-03-01 03:29:23,833] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:29:23,833] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 03:29:24,213] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 03:29:24,213] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:29:24,233] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:29:24,234] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:29:24,315] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:24,447] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:29:24,715] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:25,091] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:25,092] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:29:25,772] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:25,784] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:25,889] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:29:26,211] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:26,548] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:26,549] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:29:26,883] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:26,896] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:29:27,036] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:29:27,323] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:29:27,717] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:29:27,718] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:29:28,368] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:29:28,489] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:29:28,577] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:29:28,646] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:29:28,780] - [basetestcase:829] INFO - sleep for 30 secs. ... [2022-03-01 03:29:28,951] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 03:29:28,964] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:29:38,981] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:29:38,981] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:29:52,973] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 03:29:53,000] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:30:03,019] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:30:03,019] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:30:17,351] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 03:30:17,351] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 03:30:17,398] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:30:17,462] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:30:27,477] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:30:27,478] - [task:843] INFO - Rebalance - status: running, progress: 0.00% [2022-03-01 03:30:30,492] - [data_helper:789] INFO - Received forward map, reset vbucket map, new direct_client [2022-03-01 03:30:30,505] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:30:47,520] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:30:47,535] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.07298445701599 sec [2022-03-01 03:30:47,547] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:30:47,558] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:30:47,568] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:30:47,579] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:30:48,541] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:48,681] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:30:48,957] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:49,355] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:49,355] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:49,610] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:49,611] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:30:49,613] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:49,753] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:30:50,034] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:50,416] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:50,417] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:50,668] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:50,669] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:30:50,672] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:50,775] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:30:51,081] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:51,412] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:51,413] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:51,595] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:51,596] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:30:51,599] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:51,705] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:30:52,022] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:52,360] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:52,361] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:52,534] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:52,535] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:30:52,537] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:52,651] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:30:52,943] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:53,277] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:53,278] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:53,498] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:53,499] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 03:30:53,574] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:30:53,705] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:30:53,778] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:30:53,852] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:30:53,951] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:30:54,093] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 03:30:54,501] - [task:2328] INFO - 0 items were verified [2022-03-01 03:30:56,979] - [task:2328] INFO - 10000 items were verified [2022-03-01 03:30:56,979] - [task:2332] INFO - 10000 items were verified in 2.886091947555542 sec.the average number of ops - 3464.0 per second [2022-03-01 03:30:57,002] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:30:57,013] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:30:57,532] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:57,671] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:30:57,938] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:58,319] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:58,320] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:58,611] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:58,614] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:58,758] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:30:59,031] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:30:59,411] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:30:59,412] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:30:59,676] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:30:59,679] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:30:59,782] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:31:00,101] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:00,440] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:00,441] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:00,617] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:00,620] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:00,724] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:31:01,030] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:01,368] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:01,369] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:01,540] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:01,543] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:01,645] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:31:01,930] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:02,270] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:02,271] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:02,443] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:02,444] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:31:02,446] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:02,581] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:31:02,853] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:03,237] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:03,238] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:03,535] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:03,539] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:03,677] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:31:03,951] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:04,330] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:04,331] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:04,630] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:04,634] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:04,735] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:31:05,045] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:05,382] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:05,382] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:05,556] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:05,559] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:05,659] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:31:05,956] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:06,292] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:06,293] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:06,464] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:06,468] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:06,570] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:31:06,855] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:07,190] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:07,191] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:07,367] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:07,368] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:31:07,370] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:07,504] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:31:07,773] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:08,154] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:08,155] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:08,418] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:08,421] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:08,556] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:31:08,830] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:09,217] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:09,218] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:09,473] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:09,477] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:09,582] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:31:09,901] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:10,232] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:10,233] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:10,404] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:10,408] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:10,512] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:31:10,816] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:11,157] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:11,158] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:11,333] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:11,336] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:11,442] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:31:11,733] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:12,065] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:12,065] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:12,239] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:12,240] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:31:12,243] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:12,377] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:31:12,639] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:13,026] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:13,027] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:13,317] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:13,320] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:13,459] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:31:13,731] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:14,116] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:14,117] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:14,413] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:14,416] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:14,517] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:31:14,830] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:15,160] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:15,161] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:15,335] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:15,338] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:15,438] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:31:15,744] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:16,079] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:16,079] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:16,254] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:16,257] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:16,360] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:31:16,645] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:16,983] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:16,983] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:31:17,154] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:17,156] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:31:17,157] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:31:18,446] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 2036 [2022-03-01 03:31:18,446] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:31:18,446] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 205 [2022-03-01 03:31:18,447] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 205 [2022-03-01 03:31:18,447] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1956 [2022-03-01 03:31:18,447] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 03:31:18,447] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 205 [2022-03-01 03:31:18,447] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 205 [2022-03-01 03:31:18,447] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 1969 [2022-03-01 03:31:18,447] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 03:31:18,447] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 03:31:18,448] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 03:31:18,448] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 2014 [2022-03-01 03:31:18,448] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:31:18,448] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 03:31:18,448] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 03:31:18,448] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 2025 [2022-03-01 03:31:18,448] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:31:18,449] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 03:31:18,449] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 03:31:18,449] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:31:18,449] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 03:31:18,857] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 03:31:18,857] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:31:18,879] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:31:18,879] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:31:19,046] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:19,183] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:31:19,457] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:19,842] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:19,843] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:31:20,359] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:20,367] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:20,469] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:31:20,774] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:21,110] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:21,111] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:31:21,376] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:21,384] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:21,483] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:31:21,768] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:22,102] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:22,103] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:31:22,364] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:22,373] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:22,476] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:31:22,789] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:23,124] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:23,124] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:31:23,352] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:23,362] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:31:23,500] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:31:23,774] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:31:24,161] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:31:24,162] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:31:24,675] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:31:24,771] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:31:24,895] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:31:24,964] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:31:25,034] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:31:25,134] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:31:25,165] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 03:31:25,177] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:31:25,239] - [basetestcase:829] INFO - sleep for 30 secs. ... [2022-03-01 03:31:35,195] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:31:35,196] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:31:49,715] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 03:31:49,759] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:31:49,813] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:31:59,828] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:31:59,828] - [task:843] INFO - Rebalance - status: running, progress: 0.00% [2022-03-01 03:32:03,572] - [data_helper:789] INFO - Received forward map, reset vbucket map, new direct_client [2022-03-01 03:32:03,587] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:32:19,874] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:32:19,890] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.077046632766724 sec [2022-03-01 03:32:19,902] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:32:19,913] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:32:19,924] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:32:19,935] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:32:20,894] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:21,030] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:32:21,306] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:21,694] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:21,695] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:21,998] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:22,000] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:32:22,002] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:22,137] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:32:22,406] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:22,790] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:22,791] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:23,083] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:23,085] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:32:23,087] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:23,194] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:32:23,521] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:23,851] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:23,852] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:24,029] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:24,031] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:32:24,033] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:24,137] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:32:24,441] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:24,776] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:24,777] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:24,950] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:24,952] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:32:24,954] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:25,059] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:32:25,347] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:25,683] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:25,684] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:25,856] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:25,858] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 03:32:25,861] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:25,965] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:32:26,277] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:26,604] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:26,605] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:26,824] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:26,826] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 03:32:26,911] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:32:27,019] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:32:27,089] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:32:27,190] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:32:27,260] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:32:27,337] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:32:27,512] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 03:32:27,828] - [task:2328] INFO - 0 items were verified [2022-03-01 03:32:30,575] - [task:2328] INFO - 10000 items were verified [2022-03-01 03:32:30,575] - [task:2332] INFO - 10000 items were verified in 3.062426805496216 sec.the average number of ops - 3265.0 per second [2022-03-01 03:32:30,600] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:32:30,611] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:32:30,852] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:30,989] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:32:31,257] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:31,642] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:31,642] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:31,944] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:31,948] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:32,085] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:32:32,360] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:32,756] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:32,757] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:33,053] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:33,057] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:33,163] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:32:33,477] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:33,818] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:33,819] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:33,992] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:33,995] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:34,096] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:32:34,400] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:34,745] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:34,746] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:34,921] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:34,925] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:35,032] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:32:35,317] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:35,648] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:35,649] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:35,820] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:35,823] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:35,924] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:32:36,221] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:36,557] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:36,558] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:36,729] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:36,730] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:32:36,732] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:36,866] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:32:37,128] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:37,508] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:37,508] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:37,801] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:37,804] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:37,945] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:32:38,220] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:38,609] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:38,610] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:38,905] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:38,908] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:39,012] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:32:39,323] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:39,650] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:39,651] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:39,825] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:39,829] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:39,931] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:32:40,243] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:40,581] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:40,582] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:40,752] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:40,755] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:40,856] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:32:41,136] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:41,477] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:41,477] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:41,651] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:41,654] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:41,760] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:32:42,063] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:42,436] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:42,436] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:42,609] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:42,611] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:32:42,613] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:42,749] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:32:43,018] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:43,404] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:43,405] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:43,708] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:43,710] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:43,848] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:32:44,123] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:44,504] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:44,505] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:44,807] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:44,810] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:44,912] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:32:45,229] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:45,559] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:45,560] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:45,732] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:45,735] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:45,838] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:32:46,146] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:46,487] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:46,488] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:46,657] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:46,660] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:46,764] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:32:47,102] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:47,592] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:47,593] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:47,773] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:47,776] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:47,884] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:32:48,193] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:48,529] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:48,530] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:48,702] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:48,703] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:32:48,705] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:48,850] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:32:49,133] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:49,513] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:49,514] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:49,765] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:49,768] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:49,905] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:32:50,173] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:50,549] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:50,550] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:50,801] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:50,804] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:50,910] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:32:51,222] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:51,553] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:51,554] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:51,726] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:51,729] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:51,831] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:32:52,133] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:52,467] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:52,468] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:52,639] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:52,643] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:52,747] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:32:53,030] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:53,371] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:53,372] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:53,542] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:53,545] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:53,650] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:32:54,032] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:54,460] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:54,460] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:32:54,604] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:54,605] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:32:54,605] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:32:56,227] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 1686 [2022-03-01 03:32:56,228] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:32:56,228] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 171 [2022-03-01 03:32:56,228] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 171 [2022-03-01 03:32:56,228] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1658 [2022-03-01 03:32:56,228] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 03:32:56,228] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 171 [2022-03-01 03:32:56,228] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 171 [2022-03-01 03:32:56,228] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 1617 [2022-03-01 03:32:56,228] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 03:32:56,229] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 171 [2022-03-01 03:32:56,229] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 171 [2022-03-01 03:32:56,229] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 1675 [2022-03-01 03:32:56,229] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:32:56,229] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 171 [2022-03-01 03:32:56,229] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 171 [2022-03-01 03:32:56,229] - [rebalance_helper:370] INFO - curr_items from 172.23.105.162:8091 : 1690 [2022-03-01 03:32:56,229] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.162:8091 : 0 [2022-03-01 03:32:56,229] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.162:8091 : 170 [2022-03-01 03:32:56,229] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.162:8091 : 170 [2022-03-01 03:32:56,229] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 1674 [2022-03-01 03:32:56,230] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:32:56,230] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 170 [2022-03-01 03:32:56,230] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 170 [2022-03-01 03:32:56,230] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:32:56,230] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 03:32:56,641] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 03:32:56,641] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:32:56,665] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:32:56,665] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:32:56,959] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:57,111] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:32:57,405] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:57,781] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:57,782] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:32:58,260] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:58,269] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:58,370] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:32:58,673] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:59,012] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:59,013] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:32:59,264] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:32:59,272] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:32:59,375] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:32:59,653] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:32:59,992] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:32:59,992] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:33:00,234] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:00,241] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:00,346] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:33:00,666] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:01,001] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:01,002] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:33:01,248] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:01,256] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:01,361] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:33:01,664] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:01,991] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:01,992] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:33:02,235] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:02,243] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:02,382] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:33:02,652] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:03,039] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:03,040] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:33:03,520] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:03,567] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:33:03,567] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 7.19735503560529, 'mem_free': 2876403712, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:33:03,567] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['kv'], 'cpu_utilization': 3.701838327877109, 'mem_free': 3476144128, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:33:03,567] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['kv'], 'cpu_utilization': 3.350970017636684, 'mem_free': 3475996672, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 03:33:03,567] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['kv'], 'cpu_utilization': 2.545362903225806, 'mem_free': 3458228224, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:33:03,567] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['kv'], 'cpu_utilization': 3.371917463512833, 'mem_free': 3486887936, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 03:33:03,568] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['kv'], 'cpu_utilization': 3.784056508577195, 'mem_free': 3163684864, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:33:03,568] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:33:03,570] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:03,704] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:33:03,979] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:04,356] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:04,359] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:04,494] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:33:04,763] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:05,163] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:05,165] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:05,265] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:33:05,570] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:05,902] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:05,905] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:06,009] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:33:06,328] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:06,672] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:06,675] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:06,780] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:33:07,070] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:07,408] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:07,411] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:07,519] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:33:07,830] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:08,162] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:20,634] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #3 incremental_rebalance_in_with_ops ============== [2022-03-01 03:33:20,693] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 03:33:21,589] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 03:33:21,594] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 03:33:21,594] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 03:33:21,606] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 03:33:21,614] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 03:33:21,622] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 03:33:21,631] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 03:33:21,639] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 03:33:21,642] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 03:33:21,670] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 03:33:21,670] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 03:33:21,677] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:21,701] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 03:33:21,704] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:33:21,760] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:33:21,763] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:33:41,793] - [rest_client:1951] INFO - rebalance progress took 20.03 seconds [2022-03-01 03:33:41,793] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:33:51,837] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.105.162', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 03:33:51,844] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 03:33:51,845] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:51,851] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 03:33:51,855] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 03:33:51,855] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:51,858] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 03:33:51,861] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 03:33:51,862] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:51,865] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 03:33:51,868] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 03:33:51,868] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:51,871] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 03:33:51,874] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 03:33:51,874] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:51,876] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 03:33:51,879] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 03:33:51,880] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:51,882] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 03:33:51,882] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #3 incremental_rebalance_in_with_ops ============== [2022-03-01 03:33:51,882] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 03:33:51,883] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_3 ok ---------------------------------------------------------------------- Ran 1 test in 406.091s OK rebalance_in_with_queries (rebalance.rebalancein.RebalanceInTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_4 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_queries,nodes_in=2,blob_generator=False,value_size=1024,GROUP=IN;BASIC;P0;FROM_2_0 Test Input params: {'nodes_in': '2', 'blob_generator': 'False', 'value_size': '1024', 'GROUP': 'IN;BASIC;P0;FROM_2_0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 4, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_4'} [2022-03-01 03:33:51,906] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:52,048] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:33:52,336] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:52,719] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:52,727] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:33:52,744] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:33:52,744] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #4 rebalance_in_with_queries============== [2022-03-01 03:33:52,745] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:33:52,912] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:33:52,917] - [task:159] INFO - {'uptime': '1387', 'memoryTotal': 3947372544, 'memoryFree': 2922291200, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:33:52,924] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:33:52,924] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:33:52,924] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:33:52,940] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:33:52,940] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:33:52,940] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:33:52,958] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:33:52,967] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:33:52,968] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:33:52,972] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:33:52,972] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:33:52,972] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:33:52,972] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:33:53,100] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:33:53,102] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:53,249] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:33:53,520] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:53,901] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:53,902] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:33:53,981] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:53,981] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:33:53,986] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:53,989] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:53,997] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:33:54,040] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:33:54,045] - [task:159] INFO - {'uptime': '1383', 'memoryTotal': 3947372544, 'memoryFree': 3257364480, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:33:54,047] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:33:54,048] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 03:33:54,053] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:33:54,053] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:33:54,193] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:33:54,195] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:54,334] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:33:54,611] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:54,993] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:54,993] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:33:55,071] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:55,072] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:33:55,075] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:55,079] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:55,085] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:33:55,106] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:33:55,109] - [task:159] INFO - {'uptime': '1360', 'memoryTotal': 4201627648, 'memoryFree': 3585568768, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:33:55,111] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:33:55,111] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:33:55,114] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:33:55,115] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:33:55,173] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:33:55,175] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:55,279] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:33:55,587] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:55,921] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:55,922] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:33:55,994] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:55,994] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:33:55,997] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:56,000] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:56,004] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:33:56,018] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:33:56,021] - [task:159] INFO - {'uptime': '1366', 'memoryTotal': 4201627648, 'memoryFree': 3594838016, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:33:56,023] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:33:56,024] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:33:56,027] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:33:56,027] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:33:56,085] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:33:56,087] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:56,190] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:33:56,504] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:56,848] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:56,849] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:33:56,918] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:56,918] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:33:56,921] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:56,924] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:56,928] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:33:56,940] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:33:56,942] - [task:159] INFO - {'uptime': '1366', 'memoryTotal': 4201627648, 'memoryFree': 3611275264, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:33:56,944] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:33:56,945] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:33:56,948] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:33:56,949] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:33:57,009] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:33:57,011] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:57,116] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:33:57,415] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:57,759] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:57,759] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:33:57,828] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:57,828] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:33:57,831] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:57,834] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:57,837] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:33:57,852] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:33:57,855] - [task:159] INFO - {'uptime': '1366', 'memoryTotal': 4201627648, 'memoryFree': 3611725824, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:33:57,857] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:33:57,857] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:33:57,860] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:33:57,860] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:33:57,918] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:33:57,920] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:33:58,023] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:33:58,332] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:33:58,662] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:33:58,663] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:33:58,738] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:33:58,739] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:33:58,743] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:58,745] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:33:58,750] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:33:58,761] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:33:58,940] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:34:03,941] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:34:03,962] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:34:03,963] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:04,098] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:34:04,372] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:04,754] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:04,755] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:34:04,799] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:34:04,799] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:34:04,799] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 03:34:05,795] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 03:34:05,828] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 03:34:05,828] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 03:34:06,473] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:06,574] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:07,158] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 03:34:07,161] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #4 rebalance_in_with_queries ============== [2022-03-01 03:34:07,173] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:07,314] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:34:07,596] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:07,984] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:07,986] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:08,120] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:34:08,393] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:08,781] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:08,784] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:08,890] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:34:09,215] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:09,554] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:09,557] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:09,662] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:34:09,970] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:10,316] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:10,319] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:10,424] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:34:10,717] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:11,056] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:11,059] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:11,163] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:34:11,475] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:11,809] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:23,762] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:34:23,763] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.883885656463446, 'mem_free': 2896769024, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:34:23,763] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:34:23,763] - [basetestcase:559] INFO - Time to execute basesetup : 31.859076261520386 [2022-03-01 03:34:23,763] - [basetestcase:1445] INFO - create 1000 documents... [2022-03-01 03:34:23,811] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:24,908] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:24,911] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:24,912] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:24,913] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:34:25,219] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 03:34:25,220] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:34:25,248] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:25,386] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:34:25,650] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:26,032] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:26,033] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:34:26,983] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:34:27,017] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise'] [2022-03-01 03:34:27,027] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 03:34:27,763] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:34:27,898] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:34:28,173] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:34:28,544] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:34:28,545] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:34:28,839] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:34:28,840] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:34:29,878] - [rest_client:1073] ERROR - GET http://172.23.104.172:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2022-03-01 03:34:29,879] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:34:29,938] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:34:30,133] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:34:30,325] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:34:30,539] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:34:30,782] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?stale=ok [2022-03-01 03:34:30,898] - [task:2865] INFO - view : default_view0 was created successfully in ddoc: dev_ddoc1 [2022-03-01 03:34:30,902] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?stale=ok [2022-03-01 03:34:30,924] - [task:2865] INFO - view : default_view1 was created successfully in ddoc: dev_ddoc1 [2022-03-01 03:34:30,928] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?stale=ok [2022-03-01 03:34:30,948] - [task:2865] INFO - view : default_view2 was created successfully in ddoc: dev_ddoc1 [2022-03-01 03:34:30,952] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?stale=ok [2022-03-01 03:34:30,975] - [task:2865] INFO - view : default_view3 was created successfully in ddoc: dev_ddoc1 [2022-03-01 03:34:30,979] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?stale=ok [2022-03-01 03:34:30,998] - [task:2865] INFO - view : default_view4 was created successfully in ddoc: dev_ddoc1 [2022-03-01 03:34:32,010] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true [2022-03-01 03:34:33,023] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true [2022-03-01 03:34:34,042] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true [2022-03-01 03:34:35,073] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true [2022-03-01 03:34:36,102] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true [2022-03-01 03:34:37,131] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 03:34:38,140] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,177] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,214] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,252] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,290] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,329] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,358] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:34:38,358] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:34:38,365] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,397] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:34:38,397] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:34:38,405] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,435] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:34:38,435] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:34:38,442] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,474] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:34:38,475] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:34:38,482] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:34:38,512] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:34:38,512] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:34:38,518] - [basetestcase:829] INFO - sleep for 12 secs. ... [2022-03-01 03:34:39,513] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:34:39,529] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:34:49,545] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:34:49,545] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:35:07,427] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 03:35:07,445] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:35:17,462] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:35:17,462] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:35:34,436] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 03:35:34,436] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 03:35:34,481] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:35:34,543] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:35:44,561] - [rest_client:2012] INFO - rebalance percentage : 8.00 % [2022-03-01 03:35:44,561] - [task:843] INFO - Rebalance - status: running, progress: 8.00% [2022-03-01 03:35:44,590] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:08,996] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:09,085] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:09,177] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:09,262] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:09,351] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:09,786] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:36:09,786] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:36:19,801] - [rest_client:2012] INFO - rebalance percentage : 43.00 % [2022-03-01 03:36:19,801] - [task:843] INFO - Rebalance - status: running, progress: 43.00% [2022-03-01 03:36:19,822] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:20,048] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:36:20,048] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:36:20,055] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:20,153] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:36:20,153] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:36:20,163] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:20,269] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:36:20,269] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:36:20,284] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:36:20,382] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:36:20,383] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:36:40,404] - [rest_client:2012] INFO - rebalance percentage : 62.00 % [2022-03-01 03:36:40,405] - [task:843] INFO - Rebalance - status: running, progress: 62.00% [2022-03-01 03:37:00,446] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:37:00,458] - [task:902] INFO - rebalancing was completed with progress: 100% in 85.91444373130798 sec [2022-03-01 03:37:01,466] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:01,730] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:01,799] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:01,872] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:01,939] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:02,021] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:02,088] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:37:02,088] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:37:02,097] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:02,167] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:37:02,167] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:37:02,174] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:02,243] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:37:02,243] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:37:02,249] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:02,317] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:37:02,317] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:37:02,323] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:37:02,388] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:37:02,388] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:37:02,406] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:37:02,415] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:37:02,424] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:37:02,432] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:37:03,392] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:03,530] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:37:03,807] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:04,187] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:04,188] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:04,484] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:04,485] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:37:04,487] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:04,634] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:37:04,917] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:05,306] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:05,307] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:05,604] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:05,605] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:37:05,607] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:05,720] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:37:06,039] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:06,371] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:06,372] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:06,543] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:06,544] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:37:06,642] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:37:06,718] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:37:06,785] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:37:06,920] - [task:2319] INFO - 1000 items will be verified on default bucket [2022-03-01 03:37:07,547] - [task:2328] INFO - 0 items were verified [2022-03-01 03:37:07,701] - [task:2332] INFO - 1000 items were verified in 0.7799067497253418 sec.the average number of ops - 1282.0 per second [2022-03-01 03:37:07,721] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:37:07,731] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:37:08,559] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:08,697] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:37:08,975] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:09,353] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:09,354] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:09,607] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:09,610] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:09,746] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:37:10,029] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:10,429] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:10,430] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:10,731] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:10,734] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:10,837] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:37:11,138] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:11,473] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:11,474] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:11,649] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:11,650] - [task:972] INFO - Saw curr_items 1000 == 1000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:37:11,652] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:11,789] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:37:12,061] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:12,441] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:12,442] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:12,734] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:12,737] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:12,872] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:37:13,144] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:13,523] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:13,524] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:13,820] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:13,823] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:13,925] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:37:14,232] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:14,562] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:14,563] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:14,698] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:14,699] - [task:972] INFO - Saw vb_active_curr_items 1000 == 1000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:37:14,701] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:14,837] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:37:15,105] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:15,492] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:15,494] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:15,791] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:15,794] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:15,928] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:37:16,191] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:16,577] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:16,578] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:16,875] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:16,878] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:16,983] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:37:17,287] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:17,626] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:17,626] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:17,797] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:17,798] - [task:972] INFO - Saw vb_replica_curr_items 1000 == 1000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:37:17,800] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:17,950] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:37:18,260] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:18,656] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:18,657] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:18,960] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:18,963] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:19,132] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:37:19,512] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:20,052] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:20,053] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:20,422] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:20,425] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:20,726] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:37:21,393] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:22,153] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:22,154] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:37:22,400] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:37:22,402] - [task:972] INFO - Saw curr_items_tot 2000 == 2000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:37:22,402] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:37:23,406] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 336 [2022-03-01 03:37:23,406] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:37:23,407] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 342 [2022-03-01 03:37:23,407] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 342 [2022-03-01 03:37:23,407] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 333 [2022-03-01 03:37:23,407] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:37:23,407] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 341 [2022-03-01 03:37:23,407] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 341 [2022-03-01 03:37:23,407] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 331 [2022-03-01 03:37:23,407] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:37:23,407] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 341 [2022-03-01 03:37:23,407] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 341 [2022-03-01 03:37:23,407] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:37:23,408] - [rebalance_helper:390] INFO - sum : 1000 and sum * (replica_factor + 1) (2) : 2000 [2022-03-01 03:37:23,817] - [rebalance_helper:393] INFO - curr_items_tot from master: 2000 [2022-03-01 03:37:23,818] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:37:23,835] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:37:23,835] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 8.674637496820148, 'mem_free': 2758889472, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:37:23,836] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['kv'], 'cpu_utilization': 4.409171075837743, 'mem_free': 3438280704, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:37:23,836] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['kv'], 'cpu_utilization': 6.422250316055626, 'mem_free': 3115646976, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:37:23,836] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:37:23,838] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:23,982] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:37:24,251] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:24,630] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:24,633] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:24,768] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:37:25,040] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:25,420] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:25,423] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:25,526] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:37:25,842] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:26,183] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:26,186] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:26,300] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:37:26,617] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:26,964] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:26,968] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:27,077] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:37:27,368] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:27,703] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:27,707] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:37:27,817] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:37:28,142] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:37:28,481] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:37:41,208] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #4 rebalance_in_with_queries ============== [2022-03-01 03:37:41,256] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 03:37:42,525] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 03:37:42,530] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 03:37:42,530] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 03:37:42,542] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 03:37:42,548] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 03:37:42,554] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 03:37:42,560] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 03:37:42,565] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 03:37:42,566] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 03:37:42,590] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 03:37:42,591] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 03:37:42,597] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:37:42,617] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 03:37:42,620] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:37:42,677] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:37:42,687] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:38:02,716] - [rest_client:1951] INFO - rebalance progress took 20.04 seconds [2022-03-01 03:38:02,716] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:38:12,743] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 03:38:12,751] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 03:38:12,751] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:12,756] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 03:38:12,760] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 03:38:12,761] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:12,764] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 03:38:12,768] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 03:38:12,768] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:12,771] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 03:38:12,775] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 03:38:12,775] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:12,778] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 03:38:12,782] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 03:38:12,782] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:12,785] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 03:38:12,789] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 03:38:12,789] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:12,792] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 03:38:12,792] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #4 rebalance_in_with_queries ============== [2022-03-01 03:38:12,792] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 03:38:12,792] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_4 ok ---------------------------------------------------------------------- Ran 1 test in 260.897s OK test_rebalance_in_out_after_mutation (rebalance.rebalanceinout.RebalanceInOutTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_5 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalanceinout.RebalanceInOutTests.test_rebalance_in_out_after_mutation,nodes_in=1,nodes_init=3,replicas=1,items=10000,GROUP=IN_OUT;P0 Test Input params: {'nodes_in': '1', 'nodes_init': '3', 'replicas': '1', 'items': '10000', 'GROUP': 'IN_OUT;P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 5, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_5'} [2022-03-01 03:38:12,817] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:12,971] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:38:13,273] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:13,668] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:13,676] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:38:13,693] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:38:13,693] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #5 test_rebalance_in_out_after_mutation============== [2022-03-01 03:38:13,693] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:38:13,825] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:38:13,832] - [task:159] INFO - {'uptime': '1648', 'memoryTotal': 3947372544, 'memoryFree': 2871226368, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:38:13,841] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:38:13,841] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:38:13,842] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:38:13,854] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:38:13,854] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:38:13,854] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:38:13,874] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:38:13,886] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:38:13,886] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:38:13,889] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:38:13,889] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:38:13,889] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:38:13,889] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:38:14,008] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:38:14,010] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:14,144] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:38:14,441] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:14,850] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:14,851] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:38:14,939] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:38:14,939] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:38:14,943] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:14,947] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:14,955] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:38:14,989] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:38:14,993] - [task:159] INFO - {'uptime': '1645', 'memoryTotal': 3947372544, 'memoryFree': 3271700480, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:38:14,995] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:38:14,995] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 03:38:15,000] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:38:15,001] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:38:15,132] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:38:15,134] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:15,285] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:38:15,587] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:15,996] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:15,997] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:38:16,076] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:38:16,076] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:38:16,079] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:16,082] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:16,087] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:38:16,107] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:38:16,119] - [task:159] INFO - {'uptime': '1621', 'memoryTotal': 4201627648, 'memoryFree': 3583463424, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:38:16,130] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:38:16,130] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:38:16,143] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:38:16,143] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:38:16,205] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:38:16,207] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:16,330] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:38:16,684] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:17,051] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:17,052] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:38:17,092] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:38:17,092] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:38:17,102] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:17,112] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:17,124] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:38:17,158] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:38:17,161] - [task:159] INFO - {'uptime': '1627', 'memoryTotal': 4201627648, 'memoryFree': 3571933184, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:38:17,164] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:38:17,168] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:38:17,168] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:38:17,220] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:38:17,222] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:17,385] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:38:17,766] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:18,147] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:18,147] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:38:18,231] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:38:18,231] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:38:18,238] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:18,241] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:18,245] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:38:18,257] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:38:18,260] - [task:159] INFO - {'uptime': '1627', 'memoryTotal': 4201627648, 'memoryFree': 3588354048, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:38:18,264] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:38:18,268] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:38:18,268] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:38:18,320] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:38:18,321] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:18,485] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:38:18,818] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:19,216] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:19,217] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:38:19,299] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:38:19,299] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:38:19,310] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:19,321] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:19,327] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:38:19,340] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:38:19,343] - [task:159] INFO - {'uptime': '1627', 'memoryTotal': 4201627648, 'memoryFree': 3588341760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:38:19,347] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:38:19,350] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:38:19,350] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:38:19,402] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:38:19,404] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:38:19,523] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:38:19,853] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:38:20,231] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:38:20,232] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:38:20,268] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:38:20,268] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:38:20,271] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:20,273] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:38:20,277] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:38:20,288] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:38:20,454] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:38:25,460] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:38:25,479] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:38:26,295] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:38:26,304] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:38:36,321] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:38:36,321] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:38:52,670] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 03:38:52,686] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:39:02,702] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:39:02,702] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:39:16,485] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 03:39:16,486] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 03:39:16,495] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:39:16,566] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:39:26,581] - [rest_client:2012] INFO - rebalance percentage : 27.00 % [2022-03-01 03:39:26,581] - [task:843] INFO - Rebalance - status: running, progress: 27.00% [2022-03-01 03:39:46,614] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:39:46,624] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.057651042938232 sec [2022-03-01 03:39:46,626] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:46,762] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:39:47,046] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:47,430] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:39:47,430] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:39:47,473] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:39:47,473] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:39:47,473] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 03:39:47,653] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 03:39:47,688] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 03:39:47,688] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 03:39:48,460] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:39:48,602] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:39:48,721] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:39:48,965] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:39:49,091] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:39:49,227] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:39:49,972] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 03:39:49,975] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #5 test_rebalance_in_out_after_mutation ============== [2022-03-01 03:39:49,987] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:50,132] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:39:50,459] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:50,852] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:39:50,855] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:50,992] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:39:51,263] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:51,659] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:39:51,662] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:51,770] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:39:52,086] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:52,415] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:39:52,418] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:52,522] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:39:52,830] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:53,164] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:39:53,166] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:53,270] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:39:53,549] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:53,875] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:39:53,879] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:39:53,981] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:39:54,295] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:39:54,625] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:06,880] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:40:06,880] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 11.85015290519878, 'mem_free': 2830438400, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:40:06,880] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.031424227065383, 'mem_free': 3399901184, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:40:06,880] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 17.00258397932816, 'mem_free': 3095973888, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:40:06,880] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:40:06,880] - [basetestcase:559] INFO - Time to execute basesetup : 114.0650475025177 [2022-03-01 03:40:06,971] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:07,047] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:07,112] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:07,959] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:07,960] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:07,962] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:07,964] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:08,043] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:08,044] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:08,045] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:08,047] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:08,168] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:08,175] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:08,176] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:08,177] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:08,868] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:40:08,868] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:40:08,945] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:09,092] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:09,385] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:09,786] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:09,787] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:40:10,437] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:10,450] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:10,556] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:10,879] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:11,213] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:11,214] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:40:11,548] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:11,560] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:11,696] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:11,970] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:12,359] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:12,360] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:40:13,052] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:13,183] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:13,263] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:13,326] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:13,984] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:13,985] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:13,986] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:13,997] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:14,079] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:14,080] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:14,087] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:14,092] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:14,213] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:14,215] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:14,216] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:14,218] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:14,998] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:40:15,127] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:40:15,194] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:40:28,094] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:40:28,105] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:40:28,921] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:29,059] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:29,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:29,708] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:29,709] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:30,005] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:30,008] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:30,147] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:30,420] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:30,811] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:30,812] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:31,106] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:31,109] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:31,212] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:31,522] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:31,864] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:31,865] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:32,040] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:32,041] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:40:32,043] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:32,187] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:32,464] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:32,860] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:32,861] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:33,141] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:33,144] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:33,277] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:33,551] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:33,942] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:33,943] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:34,235] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:34,238] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:34,343] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:34,652] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:34,988] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:34,989] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:35,165] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:35,166] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:40:35,168] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:35,304] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:35,572] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:35,953] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:35,954] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:36,251] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:36,254] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:36,388] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:36,650] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:37,032] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:37,033] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:37,291] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:37,294] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:37,397] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:37,706] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:38,042] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:38,042] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:38,222] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:38,223] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:40:38,225] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:38,369] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:38,672] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:39,065] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:39,066] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:39,361] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:39,364] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:39,504] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:39,771] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:40,156] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:40,157] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:40,447] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:40,450] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:40,552] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:40,860] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:41,194] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:41,195] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:41,367] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:41,368] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 03:40:41,386] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:40:41,396] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:40:42,371] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:42,505] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:42,775] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:43,151] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:43,152] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:43,451] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:43,452] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:40:43,455] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:43,592] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:43,867] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:44,266] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:44,267] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:44,603] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:44,604] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:40:44,606] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:44,711] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:45,039] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:45,377] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:45,378] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:40:45,544] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:45,546] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:40:45,548] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:45,681] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:45,952] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:46,331] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:46,332] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:40:47,013] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:47,026] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:47,160] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:47,428] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:47,820] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:47,821] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:40:48,472] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:48,486] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:48,591] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:48,900] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:49,233] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:49,234] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:40:49,573] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:49,683] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:49,827] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:50,108] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:50,492] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:50,492] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:40:50,794] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:50,795] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:40:51,051] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:51,055] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:51,193] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:51,464] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:51,848] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:51,849] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:40:52,145] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:52,147] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:40:52,442] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:52,446] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:52,547] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:52,857] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:53,184] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:53,184] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:40:53,354] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:53,356] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:40:53,487] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:53,491] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:53,628] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:53,893] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:54,275] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:54,276] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:40:54,694] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:54,705] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:54,843] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:55,116] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:55,494] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:55,495] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:40:55,920] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:55,931] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:56,036] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:40:56,339] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:56,671] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:56,672] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:40:56,895] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:40:56,978] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:40:56,989] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:40:57,004] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:57,140] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:40:57,408] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:40:57,792] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:40:57,794] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:40:59,216] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:40:59,462] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:40:59,595] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:40:59,869] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:41:00,258] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:41:00,259] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:41:01,810] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:41:02,070] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:41:02,172] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:41:02,480] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:41:02,815] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:41:02,816] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:41:03,470] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:41:03,690] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:41:03,826] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:41:04,109] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:41:04,487] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:41:04,488] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:41:05,906] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:41:06,155] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:41:06,289] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:41:06,567] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:41:06,945] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:41:06,946] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:41:08,404] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:41:08,655] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:41:08,756] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:41:09,068] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:41:09,399] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:41:09,399] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:41:10,051] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:41:10,260] - [basetestcase:1984] INFO - Begin Verification for Active Vs Replica [2022-03-01 03:41:10,307] - [basetestcase:1990] INFO - End Verification for Active Vs Replica [2022-03-01 03:41:10,332] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:41:20,347] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-03-01 03:41:20,347] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:41:34,301] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 03:41:34,350] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 03:41:34,375] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.158', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:41:34,444] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:41:34,447] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:41:44,461] - [rest_client:2012] INFO - rebalance percentage : 29.00 % [2022-03-01 03:41:54,476] - [rest_client:2012] INFO - rebalance percentage : 43.00 % [2022-03-01 03:42:04,490] - [rest_client:2012] INFO - rebalance percentage : 62.00 % [2022-03-01 03:42:14,506] - [rest_client:2012] INFO - rebalance percentage : 62.00 % [2022-03-01 03:42:34,534] - [rest_client:1951] INFO - rebalance progress took 60.09 seconds [2022-03-01 03:42:34,534] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:42:44,544] - [rebalance_base:112] INFO - successfully rebalanced cluster True [2022-03-01 03:42:44,564] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:42:44,574] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:42:44,678] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:44,814] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:42:45,090] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:45,486] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:45,487] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:45,815] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:45,818] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:45,959] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:42:46,230] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:46,609] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:46,610] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:46,898] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:46,902] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:47,005] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:42:47,311] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:47,642] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:47,643] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:47,776] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:47,777] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:42:47,779] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:47,914] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:42:48,185] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:48,572] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:48,573] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:48,825] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:48,829] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:48,970] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:42:49,245] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:49,630] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:49,631] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:49,922] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:49,925] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:50,032] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:42:50,342] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:50,681] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:50,682] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:50,859] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:50,860] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:42:50,862] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:50,996] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:42:51,263] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:51,657] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:51,658] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:51,911] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:51,914] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:52,050] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:42:52,316] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:52,695] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:52,696] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:52,996] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:53,000] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:53,101] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:42:53,403] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:53,748] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:53,749] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:53,922] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:53,924] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:42:53,926] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:54,060] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:42:54,328] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:54,711] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:54,712] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:55,004] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:55,007] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:55,149] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:42:55,424] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:55,809] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:55,810] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:56,106] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:56,110] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:56,214] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:42:56,512] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:56,845] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:56,846] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:57,020] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:57,021] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:42:57,041] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:42:57,051] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:42:57,061] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:42:57,073] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:42:58,026] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:58,162] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:42:58,431] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:58,813] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:58,814] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:42:59,075] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:42:59,077] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:42:59,079] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:42:59,210] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:42:59,490] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:42:59,881] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:42:59,882] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:43:02,498] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:02,510] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:43:02,512] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:02,651] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:02,926] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:03,317] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:03,318] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:03,620] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:03,621] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:43:03,623] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:03,754] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:04,032] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:04,411] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:04,411] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:43:06,972] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:06,982] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:43:06,984] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:07,087] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:07,386] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:07,718] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:07,719] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:07,891] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:07,893] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:43:07,895] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:07,995] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:08,296] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:08,619] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:08,620] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:43:09,699] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:09,711] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:43:09,782] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:43:09,859] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:43:09,964] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:43:10,086] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 03:43:10,713] - [task:2328] INFO - 0 items were verified [2022-03-01 03:43:12,857] - [task:2328] INFO - 10000 items were verified [2022-03-01 03:43:12,857] - [task:2332] INFO - 10000 items were verified in 2.770214080810547 sec.the average number of ops - 3609.0 per second [2022-03-01 03:43:12,877] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:43:12,888] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:43:13,740] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:13,877] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:14,162] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:14,560] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:14,561] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:14,826] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:14,829] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:14,967] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:15,234] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:15,617] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:15,618] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:15,912] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:15,915] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:16,016] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:16,323] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:16,667] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:16,667] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:16,842] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:16,843] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:43:16,845] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:16,981] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:17,250] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:17,631] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:17,632] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:17,932] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:17,935] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:18,069] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:18,343] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:18,730] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:18,731] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:18,994] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:18,997] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:19,098] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:19,399] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:19,732] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:19,733] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:19,909] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:19,910] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:43:19,912] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:20,053] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:20,348] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:20,737] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:20,738] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:21,036] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:21,039] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:21,174] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:21,442] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:21,821] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:21,821] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:22,121] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:22,124] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:22,229] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:22,535] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:22,871] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:22,872] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:23,043] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:23,045] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:43:23,047] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:23,181] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:23,452] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:23,842] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:23,843] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:24,101] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:24,104] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:24,240] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:24,503] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:24,886] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:24,887] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:25,175] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:25,178] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:25,282] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:25,581] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:25,914] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:25,914] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:43:26,086] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:26,087] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.126:8091',default bucket [2022-03-01 03:43:26,088] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:43:27,036] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 3318 [2022-03-01 03:43:27,037] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:43:27,037] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 342 [2022-03-01 03:43:27,037] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 342 [2022-03-01 03:43:27,037] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 3340 [2022-03-01 03:43:27,037] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 03:43:27,037] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 341 [2022-03-01 03:43:27,038] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 341 [2022-03-01 03:43:27,038] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 3342 [2022-03-01 03:43:27,038] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:43:27,038] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 341 [2022-03-01 03:43:27,038] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 341 [2022-03-01 03:43:27,038] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:43:27,038] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 03:43:27,429] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 03:43:27,429] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:43:27,431] - [basetestcase:2301] INFO - Begin Verification for failovers logs comparison [2022-03-01 03:43:27,433] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:27,570] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:27,841] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:28,222] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:28,223] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:43:28,518] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:28,520] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:43:28,781] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:28,785] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:28,922] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:29,199] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:29,580] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:29,581] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:43:29,878] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:29,879] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:43:30,182] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:30,186] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:30,289] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:30,599] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:30,947] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:30,947] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:43:31,122] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:31,123] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:43:31,265] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:31,269] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:31,409] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:31,703] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:32,097] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:32,098] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:43:32,481] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:32,492] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:32,627] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:32,898] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:33,281] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:33,281] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:43:33,712] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:33,723] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:33,825] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:34,127] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:34,464] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:34,465] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:43:34,696] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:34,765] - [basetestcase:2308] INFO - End Verification for Failovers logs comparison [2022-03-01 03:43:34,765] - [basetestcase:1863] INFO - Begin Verification for vbucket sequence numbers comparison [2022-03-01 03:43:34,767] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:34,907] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:35,178] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:35,555] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:35,555] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:43:36,198] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:36,211] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:36,347] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:36,624] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:37,011] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:37,012] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:43:37,728] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:37,740] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:37,842] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:38,140] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:38,473] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:38,474] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:43:38,810] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:43:38,875] - [basetestcase:1888] INFO - End Verification for vbucket sequence numbers comparison [2022-03-01 03:43:38,876] - [basetestcase:2105] INFO - Begin Verification for data comparison [2022-03-01 03:43:38,893] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:39,035] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:39,302] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:39,682] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:39,683] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:43:41,117] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:43:41,364] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:41,501] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:41,774] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:42,160] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:42,160] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:43:43,599] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:43:43,838] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:43,946] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:44,259] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:44,600] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:44,601] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:43:45,272] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:43:45,505] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:45,641] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:45,915] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:46,304] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:46,305] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:43:47,727] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:43:47,973] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:48,106] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:48,378] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:48,769] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:48,770] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:43:50,187] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:43:50,431] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:50,536] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:50,838] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:51,172] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:51,173] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:43:51,833] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:43:52,055] - [basetestcase:2110] INFO - Comparing :: Prev vs Current :: Active and Replica [2022-03-01 03:43:52,156] - [basetestcase:2123] INFO - Comparing :: Current :: Active and Replica [2022-03-01 03:43:52,201] - [basetestcase:2127] INFO - End Verification for data comparison [2022-03-01 03:43:52,249] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 03:43:52,260] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:43:52,270] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:43:52,280] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:43:52,290] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:43:52,290] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 03:43:52,315] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:43:52,315] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 10.56642113284227, 'mem_free': 2803101696, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:43:52,315] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['kv'], 'cpu_utilization': 3.929471032745592, 'mem_free': 3472220160, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:43:52,316] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 9.850595087363889, 'mem_free': 3033886720, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:43:52,316] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:43:52,318] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:52,449] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:43:52,732] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:53,117] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:53,120] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:53,254] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:43:53,532] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:53,924] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:53,927] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:54,036] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:43:54,347] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:54,679] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:54,682] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:54,783] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:43:55,093] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:55,431] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:55,434] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:55,538] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:43:55,823] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:56,153] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:43:56,155] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:43:56,260] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:43:56,571] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:43:56,904] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:44:08,923] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #5 test_rebalance_in_out_after_mutation ============== [2022-03-01 03:44:09,009] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 03:44:10,705] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 03:44:10,709] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 03:44:10,709] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 03:44:10,721] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 03:44:10,728] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 03:44:10,736] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 03:44:10,741] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 03:44:10,746] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 03:44:10,747] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 03:44:10,770] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 03:44:10,772] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 03:44:10,778] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:44:10,798] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 03:44:10,800] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:44:10,861] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:44:10,864] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:44:20,878] - [rest_client:2012] INFO - rebalance percentage : 66.00 % [2022-03-01 03:44:30,893] - [rest_client:2012] INFO - rebalance percentage : 66.00 % [2022-03-01 03:44:50,919] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 03:44:50,920] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:45:00,944] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 03:45:00,952] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 03:45:00,952] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:00,957] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 03:45:00,962] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 03:45:00,963] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:00,967] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 03:45:00,971] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 03:45:00,971] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:00,974] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 03:45:00,977] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 03:45:00,977] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:00,980] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 03:45:00,984] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 03:45:00,984] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:00,986] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 03:45:00,990] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 03:45:00,990] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:00,993] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 03:45:00,993] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #5 test_rebalance_in_out_after_mutation ============== [2022-03-01 03:45:00,993] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 03:45:00,994] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_5 ok ---------------------------------------------------------------------- Ran 1 test in 408.187s OK rebalance_out_after_ops (rebalance.rebalanceout.RebalanceOutTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_6 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_after_ops,nodes_out=1,replicas=1,items=10000,GROUP=OUT;P0 Test Input params: {'nodes_out': '1', 'replicas': '1', 'items': '10000', 'GROUP': 'OUT;P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 6, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_6'} [2022-03-01 03:45:01,024] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:01,166] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:45:01,451] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:01,835] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:01,842] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:45:01,860] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:45:01,860] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #6 rebalance_out_after_ops============== [2022-03-01 03:45:01,861] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:45:02,030] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:45:02,036] - [task:159] INFO - {'uptime': '2056', 'memoryTotal': 3947372544, 'memoryFree': 2874241024, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:45:02,043] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:45:02,043] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:45:02,043] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:45:02,058] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:45:02,058] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:45:02,059] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:45:02,072] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:45:02,087] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:45:02,087] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:45:02,090] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:45:02,090] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:45:02,090] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:45:02,090] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:45:02,222] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:45:02,224] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:02,364] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:45:02,624] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:03,009] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:03,010] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:45:03,088] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:45:03,089] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:45:03,092] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:03,095] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:03,103] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:45:03,129] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:45:03,133] - [task:159] INFO - {'uptime': '2054', 'memoryTotal': 3947372544, 'memoryFree': 3267584000, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:45:03,136] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:45:03,136] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 03:45:03,142] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:45:03,142] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:45:03,271] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:45:03,273] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:03,422] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:45:03,718] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:04,103] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:04,104] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:45:04,183] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:45:04,184] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:45:04,189] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:04,191] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:04,197] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:45:04,219] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:45:04,222] - [task:159] INFO - {'uptime': '2033', 'memoryTotal': 4201627648, 'memoryFree': 3579871232, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:45:04,223] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:45:04,224] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:45:04,227] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:45:04,227] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:45:04,293] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:45:04,295] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:04,402] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:45:04,718] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:05,053] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:05,053] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:45:05,124] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:45:05,124] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:45:05,127] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:05,129] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:05,133] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:45:05,148] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:45:05,150] - [task:159] INFO - {'uptime': '2031', 'memoryTotal': 4201627648, 'memoryFree': 3598630912, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:45:05,152] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:45:05,153] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:45:05,156] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:45:05,156] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:45:05,212] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:45:05,214] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:05,322] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:45:05,637] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:05,982] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:05,982] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:45:06,051] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:45:06,051] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:45:06,055] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:06,057] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:06,061] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:45:06,072] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:45:06,076] - [task:159] INFO - {'uptime': '2034', 'memoryTotal': 4201627648, 'memoryFree': 3553525760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:45:06,079] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:45:06,083] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:45:06,083] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:45:06,134] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:45:06,136] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:06,238] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:45:06,529] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:06,858] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:06,859] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:45:06,933] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:45:06,933] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:45:06,936] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:06,938] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:06,943] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:45:06,958] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:45:06,961] - [task:159] INFO - {'uptime': '2034', 'memoryTotal': 4201627648, 'memoryFree': 3561197568, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:45:06,965] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:45:06,969] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:45:06,969] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:45:07,021] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:45:07,023] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:45:07,125] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:45:07,446] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:45:07,785] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:45:07,786] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:45:07,856] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:45:07,856] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:45:07,859] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:07,861] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:45:07,864] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:45:07,875] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:45:08,048] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:45:13,053] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:45:13,072] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:45:13,881] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:45:13,891] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:45:23,907] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:45:23,908] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:45:40,731] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 03:45:40,746] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:45:50,763] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:45:50,764] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:46:04,477] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 03:46:04,499] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:46:14,520] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:46:14,520] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:46:28,247] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 03:46:28,267] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:46:38,284] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:46:38,284] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:46:51,838] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 03:46:51,860] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:47:01,879] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:47:01,880] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:47:15,297] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 03:47:15,297] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 03:47:15,298] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 03:47:15,298] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 03:47:15,298] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 03:47:15,309] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:47:15,365] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:47:25,380] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:47:25,395] - [task:902] INFO - rebalancing was completed with progress: 100% in 10.029102802276611 sec [2022-03-01 03:47:25,397] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:25,536] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:47:25,807] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:26,183] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:26,184] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:47:26,223] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:26,223] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:47:26,223] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 03:47:26,430] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 03:47:26,468] - [rest_client:3014] INFO - 0.04 seconds to create bucket default [2022-03-01 03:47:26,468] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 03:47:27,284] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:27,469] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:27,605] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:27,735] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:27,902] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:28,038] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:28,394] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:28,581] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:28,722] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:28,920] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:29,027] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:29,148] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:29,922] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 03:47:29,924] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #6 rebalance_out_after_ops ============== [2022-03-01 03:47:29,937] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:30,076] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:47:30,367] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:30,755] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:30,758] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:30,896] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:47:31,169] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:31,558] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:31,562] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:31,664] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:47:31,978] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:32,308] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:32,311] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:32,414] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:47:32,718] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:33,054] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:33,056] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:33,170] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:47:33,495] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:33,896] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:33,899] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:34,065] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:47:34,454] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:34,883] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:47,048] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:47:47,048] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 11.58055484856198, 'mem_free': 2836668416, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:47:47,048] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.593963986812072, 'mem_free': 3395174400, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:47:47,048] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.404066073697586, 'mem_free': 3360993280, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 03:47:47,048] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.632694248234107, 'mem_free': 3390201856, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:47:47,048] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.736040609137055, 'mem_free': 3363385344, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 03:47:47,048] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 16.58962506420134, 'mem_free': 3059490816, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:47:47,049] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:47:47,049] - [basetestcase:559] INFO - Time to execute basesetup : 166.02679800987244 [2022-03-01 03:47:47,163] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:47,273] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:47,354] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:47,469] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:47,549] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:47,636] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:48,161] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:48,161] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:48,161] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:48,162] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:47:48,290] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:48,290] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:48,293] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:48,294] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:47:48,449] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:48,450] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:48,451] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:48,451] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:47:48,534] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:48,534] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:48,535] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:48,537] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:47:48,623] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:48,625] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:48,626] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:48,627] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:47:48,772] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:48,774] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:48,775] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:48,775] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:47:49,546] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:47:49,547] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:47:49,778] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:49,918] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:47:50,189] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:50,570] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:50,571] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:47:51,027] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:51,036] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:51,141] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:47:51,444] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:51,781] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:51,782] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:47:52,029] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:52,037] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:52,141] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:47:52,427] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:52,762] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:52,763] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:47:53,019] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:53,027] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:53,133] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:47:53,444] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:53,773] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:53,774] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:47:54,024] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:54,032] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:54,138] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:47:54,445] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:54,773] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:54,774] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:47:55,017] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:55,025] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:55,162] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:47:55,443] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:55,822] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:55,822] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:47:56,260] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:56,296] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:47:56,307] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:47:57,055] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:57,202] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:47:57,487] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:57,864] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:57,864] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:47:58,155] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:58,158] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:58,298] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:47:58,562] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:58,952] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:58,953] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:47:59,214] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:47:59,217] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:47:59,320] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:47:59,625] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:47:59,954] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:47:59,955] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:00,125] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:00,128] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:00,232] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:00,537] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:00,873] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:00,874] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:01,051] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:01,054] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:01,156] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:01,438] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:01,773] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:01,774] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:01,944] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:01,948] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:02,050] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:02,354] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:02,687] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:02,688] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:02,863] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:02,865] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:48:02,867] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:03,003] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:03,269] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:03,652] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:03,653] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:03,944] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:03,948] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:04,088] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:04,371] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:04,751] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:04,751] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:05,043] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:05,046] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:05,150] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:05,462] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:05,797] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:05,798] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:05,971] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:05,974] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:06,078] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:06,388] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:06,728] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:06,729] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:06,902] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:06,906] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:07,011] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:07,289] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:07,624] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:07,625] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:07,796] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:07,799] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:07,903] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:08,212] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:08,547] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:08,548] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:08,727] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:08,728] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:48:08,730] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:08,867] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:09,145] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:09,533] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:09,534] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:09,827] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:09,830] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:09,965] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:10,238] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:10,627] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:10,628] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:10,921] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:10,924] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:11,024] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:11,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:11,658] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:11,659] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:11,833] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:11,836] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:11,940] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:12,242] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:12,581] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:12,582] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:12,752] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:12,755] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:12,858] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:13,148] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:13,492] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:13,493] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:13,673] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:13,677] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:13,781] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:14,091] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:14,430] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:14,431] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:14,608] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:14,610] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:48:14,612] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:14,746] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:15,015] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:15,393] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:15,394] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:15,695] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:15,698] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:15,835] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:16,116] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:16,498] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:16,498] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:16,789] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:16,793] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:16,897] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:17,205] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:17,537] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:17,538] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:17,709] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:17,712] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:17,813] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:18,108] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:18,448] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:18,449] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:18,620] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:18,623] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:18,725] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:19,012] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:19,347] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:19,347] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:19,518] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:19,521] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:19,622] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:19,929] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:20,256] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:20,256] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:20,427] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:20,428] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 03:48:20,452] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:48:20,464] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:48:21,431] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:21,571] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:21,839] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:22,222] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:22,223] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:22,516] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:22,518] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:48:22,520] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:22,653] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:22,919] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:23,299] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:23,300] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:23,599] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:23,601] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:48:23,603] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:23,708] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:24,028] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:24,364] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:24,365] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:24,537] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:24,539] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:48:24,541] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:24,644] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:24,956] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:25,302] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:25,303] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:25,478] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:25,479] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:48:25,481] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:25,587] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:25,884] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:26,219] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:26,220] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:26,392] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:26,394] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 03:48:26,396] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:26,501] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:26,821] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:27,158] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:27,159] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:48:27,329] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:27,331] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 03:48:27,333] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:27,474] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:27,745] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:28,125] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:28,126] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:28,401] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:28,402] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:28,677] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:28,680] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:28,817] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:29,082] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:29,464] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:29,464] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:29,712] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:29,713] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:29,955] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:29,957] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:30,064] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:30,395] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:30,734] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:30,735] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:30,902] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:30,903] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:31,026] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:31,029] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:31,135] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:31,448] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:31,791] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:31,792] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:31,965] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:31,966] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:32,131] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:32,134] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:32,240] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:32,547] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:32,891] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:32,893] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:33,059] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:33,060] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:33,226] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:33,229] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:33,334] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:33,651] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:33,978] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:33,979] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:34,141] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:34,142] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:48:34,270] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:34,273] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:34,409] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:34,684] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:35,067] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:35,068] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:48:35,404] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:35,410] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:35,547] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:35,819] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:36,206] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:36,207] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:48:36,562] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:36,569] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:36,672] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:36,975] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:37,306] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:37,307] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:48:37,501] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:37,507] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:37,616] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:37,921] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:38,253] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:38,253] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:48:38,450] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:38,456] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:38,559] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:38,844] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:39,190] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:39,191] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:48:39,384] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:39,390] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:39,499] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:39,810] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:40,140] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:40,141] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:48:40,335] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:40,398] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:40,533] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:40,835] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:41,222] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:41,223] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:48:41,692] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:41,700] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:41,838] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:42,108] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:42,495] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:42,496] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:48:42,963] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:42,971] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:43,075] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:43,390] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:43,719] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:43,720] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:48:43,969] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:43,977] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:44,080] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:44,391] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:44,726] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:44,727] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:48:44,969] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:44,976] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:45,079] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:45,358] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:45,693] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:45,693] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:48:45,932] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:45,939] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:46,041] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:46,349] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:46,683] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:46,684] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:48:46,927] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:48:47,008] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:48:47,019] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:48:47,034] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:47,182] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:48:47,448] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:47,830] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:47,832] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:48:48,987] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:48:49,227] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:49,371] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:48:49,656] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:50,073] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:50,074] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:48:51,215] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:48:51,492] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:51,598] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:48:51,919] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:52,260] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:52,261] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:48:52,832] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:48:53,039] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:53,147] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:48:53,460] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:53,799] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:53,800] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:48:54,397] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:48:54,592] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:54,696] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:48:54,983] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:55,316] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:55,317] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:48:55,940] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:48:56,101] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:48:56,210] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:48:56,518] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:48:56,848] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:48:56,849] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:48:57,499] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:48:58,418] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.162', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:48:58,476] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:49:08,491] - [rest_client:2012] INFO - rebalance percentage : 25.00 % [2022-03-01 03:49:08,492] - [task:843] INFO - Rebalance - status: running, progress: 25.00% [2022-03-01 03:49:28,538] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:49:28,538] - [task:843] INFO - Rebalance - status: running, progress: 50.00% [2022-03-01 03:49:48,575] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:49:48,599] - [task:902] INFO - rebalancing was completed with progress: 100% in 50.12196707725525 sec [2022-03-01 03:49:48,612] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:49:48,622] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:49:49,602] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:49,740] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:49:50,021] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:50,398] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:50,399] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:50,695] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:50,698] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:50,837] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:49:51,112] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:51,501] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:51,502] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:51,808] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:51,810] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:51,919] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:49:52,235] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:52,566] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:52,567] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:52,743] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:52,746] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:52,849] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:49:53,160] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:53,494] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:53,494] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:53,666] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:53,669] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:53,774] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:49:54,067] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:54,408] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:54,409] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:54,581] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:54,583] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:49:54,585] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:54,725] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:49:54,998] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:55,381] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:55,382] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:55,646] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:55,649] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:55,789] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:49:56,064] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:56,443] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:56,444] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:56,751] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:56,754] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:56,861] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:49:57,178] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:57,518] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:57,519] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:57,693] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:57,697] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:57,800] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:49:58,105] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:58,444] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:58,444] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:58,619] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:58,622] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:58,723] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:49:59,008] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:49:59,345] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:49:59,346] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:49:59,513] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:49:59,515] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:49:59,517] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:49:59,653] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:49:59,933] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:00,312] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:00,312] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:00,570] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:00,573] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:00,711] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:00,988] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:01,377] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:01,378] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:01,645] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:01,649] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:01,750] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:02,065] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:02,397] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:02,398] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:02,571] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:02,574] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:02,679] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:02,990] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:03,325] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:03,325] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:03,490] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:03,492] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:03,595] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:03,872] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:04,203] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:04,204] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:04,371] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:04,372] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:50:04,375] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:04,509] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:04,785] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:05,166] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:05,167] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:05,420] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:05,423] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:05,560] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:05,826] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:06,214] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:06,214] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:06,479] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:06,482] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:06,587] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:06,903] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:07,236] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:07,236] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:07,413] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:07,416] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:07,518] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:07,821] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:08,163] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:08,164] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:08,332] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:08,335] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:08,443] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:08,731] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:09,074] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:09,074] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:09,249] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:09,250] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:50:09,276] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:50:09,287] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:50:09,299] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:50:09,311] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:50:10,255] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:10,392] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:10,664] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:11,049] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:11,050] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:11,301] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:11,302] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:50:11,304] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:11,440] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:11,721] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:12,099] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:12,100] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:50:13,779] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:13,786] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:50:13,788] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:13,926] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:14,209] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:14,590] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:14,590] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:14,888] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:14,889] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:50:14,891] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:15,035] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:15,301] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:15,684] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:15,685] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:50:17,324] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:17,331] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:50:17,333] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:17,437] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:17,753] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:18,086] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:18,086] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:18,257] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:18,258] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:50:18,260] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:18,361] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:18,670] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:19,006] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:19,007] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:50:19,747] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:19,755] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:50:19,757] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:19,859] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:20,172] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:20,513] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:20,514] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:20,690] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:20,692] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:50:20,694] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:20,796] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:21,110] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:21,448] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:21,449] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:50:22,185] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:22,192] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:50:22,195] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:22,298] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:22,583] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:22,919] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:22,919] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:23,092] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:23,094] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 03:50:23,096] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:23,197] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:23,474] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:23,809] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:23,810] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 03:50:24,524] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:24,532] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 03:50:24,663] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:50:24,767] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:50:24,838] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:50:24,958] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:50:25,042] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:50:25,185] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 03:50:25,534] - [task:2328] INFO - 0 items were verified [2022-03-01 03:50:27,945] - [task:2328] INFO - 10000 items were verified [2022-03-01 03:50:27,945] - [task:2332] INFO - 10000 items were verified in 2.759929895401001 sec.the average number of ops - 3623.0 per second [2022-03-01 03:50:27,968] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:50:27,979] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:50:28,549] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:28,687] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:28,957] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:29,339] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:29,339] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:29,649] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:29,652] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:29,787] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:30,053] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:30,433] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:30,434] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:30,699] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:30,703] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:30,806] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:31,117] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:31,446] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:31,448] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:31,621] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:31,625] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:31,727] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:32,032] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:32,369] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:32,370] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:32,540] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:32,543] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:32,646] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:32,933] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:33,264] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:33,265] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:33,436] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:33,437] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:50:33,439] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:33,577] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:33,850] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:34,229] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:34,230] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:34,522] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:34,526] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:34,663] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:34,960] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:35,345] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:35,346] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:35,637] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:35,640] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:35,749] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:36,056] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:36,385] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:36,387] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:36,559] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:36,563] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:36,662] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:36,959] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:37,289] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:37,290] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:37,463] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:37,466] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:37,566] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:37,850] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:38,188] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:38,189] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:38,359] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:38,360] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:50:38,363] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:38,497] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:38,769] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:39,151] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:39,152] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:39,443] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:39,446] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:39,581] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:39,848] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:40,232] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:40,233] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:40,524] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:40,527] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:40,628] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:40,937] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:41,264] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:41,265] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:41,436] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:41,439] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:41,539] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:41,845] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:42,185] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:42,186] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:42,358] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:42,361] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:42,464] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:42,748] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:43,079] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:43,080] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:43,252] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:43,253] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:50:43,255] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:43,407] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:43,676] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:44,055] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:44,056] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:44,353] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:44,357] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:44,491] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:44,767] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:45,152] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:45,153] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:45,440] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:45,443] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:45,544] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:45,854] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:46,186] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:46,187] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:46,359] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:46,362] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:46,464] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:46,814] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:47,155] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:47,156] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:47,333] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:47,336] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:47,436] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:47,717] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:48,044] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:48,044] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:50:48,215] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:48,217] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:50:48,217] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:50:49,473] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 2049 [2022-03-01 03:50:49,474] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:50:49,474] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 205 [2022-03-01 03:50:49,474] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 205 [2022-03-01 03:50:49,474] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1993 [2022-03-01 03:50:49,474] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 03:50:49,474] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 205 [2022-03-01 03:50:49,474] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 205 [2022-03-01 03:50:49,474] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 1924 [2022-03-01 03:50:49,474] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 03:50:49,475] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 03:50:49,475] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 03:50:49,475] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 2031 [2022-03-01 03:50:49,475] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:50:49,475] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 03:50:49,475] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 03:50:49,475] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 2003 [2022-03-01 03:50:49,475] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:50:49,475] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 03:50:49,475] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 03:50:49,476] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:50:49,476] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 03:50:49,876] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 03:50:49,876] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:50:49,878] - [basetestcase:2301] INFO - Begin Verification for failovers logs comparison [2022-03-01 03:50:49,879] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:50,021] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:50,290] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:50,661] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:50,662] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:50,950] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:50,951] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:51,201] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:51,204] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:51,341] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:51,607] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:51,994] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:51,995] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:52,248] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:52,250] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:52,490] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:52,493] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:52,598] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:52,907] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:53,240] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:53,241] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:53,402] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:53,403] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:53,525] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:53,527] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:53,638] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:53,950] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:54,290] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:54,291] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:54,457] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:54,457] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:54,636] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:54,639] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:54,742] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:50:55,029] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:55,363] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:55,363] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:55,530] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:55,531] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:50:55,700] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:55,703] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:55,839] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:50:56,107] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:56,489] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:56,490] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:50:56,863] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:56,870] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:57,011] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:50:57,291] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:57,673] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:57,674] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:50:58,031] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:58,039] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:58,139] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:50:58,453] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:58,784] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:58,785] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:50:58,985] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:58,992] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:50:59,096] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:50:59,400] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:50:59,759] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:50:59,759] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:50:59,961] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:50:59,968] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:51:00,074] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:51:00,356] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:51:00,695] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:51:00,696] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 03:51:00,897] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:51:00,961] - [basetestcase:2308] INFO - End Verification for Failovers logs comparison [2022-03-01 03:51:00,961] - [basetestcase:1863] INFO - Begin Verification for vbucket sequence numbers comparison [2022-03-01 03:51:00,962] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:51:01,097] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:51:01,373] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:51:01,753] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:51:01,754] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:51:02,279] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:51:02,287] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:51:02,425] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:51:02,695] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:51:03,088] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:51:03,090] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:51:03,606] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:51:03,616] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:51:03,722] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:51:04,036] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:51:04,373] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:51:04,374] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:51:04,637] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:51:04,646] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:51:04,750] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:51:05,054] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:51:05,393] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:51:05,394] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:51:05,656] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:51:05,665] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:51:05,768] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:51:06,057] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:51:06,399] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:51:06,400] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:51:06,673] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:51:06,739] - [basetestcase:1888] INFO - End Verification for vbucket sequence numbers comparison [2022-03-01 03:51:06,739] - [basetestcase:829] INFO - sleep for 60 secs. ... [2022-03-01 03:52:06,799] - [basetestcase:2135] INFO - Begin Verification for data comparison [2022-03-01 03:52:06,827] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:52:06,838] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:52:06,853] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:06,991] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:52:07,271] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:07,656] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:07,657] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:52:08,843] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:52:09,076] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:09,214] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:52:09,482] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:09,864] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:09,865] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:52:11,039] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:52:11,273] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:11,376] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:52:11,692] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:12,023] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:12,024] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:52:12,591] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:52:12,795] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:12,899] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:52:13,196] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:13,539] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:13,540] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:52:14,120] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:52:14,323] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:14,427] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:52:14,710] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:15,043] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:15,044] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 03:52:15,614] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 03:52:15,869] - [basetestcase:2144] INFO - End Verification for data comparison [2022-03-01 03:52:15,922] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 03:52:15,933] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:52:15,943] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:52:15,956] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:52:15,971] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:52:15,973] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:16,112] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:52:16,373] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:16,751] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:16,752] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:17,039] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:17,040] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:17,289] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:17,292] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:17,393] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:52:17,692] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:18,032] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:18,032] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:18,199] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:18,200] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:18,328] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:18,331] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:18,433] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:52:18,719] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:19,057] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:19,058] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:19,224] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:19,225] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:19,351] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:19,353] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:19,456] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:52:19,767] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:20,097] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:20,098] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:20,263] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:20,263] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:20,387] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:20,390] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:20,527] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:52:20,807] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:21,189] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:21,190] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:21,473] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:21,474] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 03:52:21,726] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:52:21,727] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 03:52:21,779] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 03:52:21,794] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:52:21,794] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.883413766827534, 'mem_free': 2800590848, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:52:21,794] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.971543691765298, 'mem_free': 3345231872, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:52:21,794] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 0.807061790668348, 'mem_free': 3340951552, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 03:52:21,794] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.447667087011349, 'mem_free': 3350908928, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:52:21,795] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.409931593615404, 'mem_free': 3012558848, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:52:21,795] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:52:21,797] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:21,932] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:52:22,213] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:22,593] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:22,596] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:22,731] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:52:23,006] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:23,395] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:23,398] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:23,501] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:52:23,815] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:24,149] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:24,152] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:24,258] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:52:24,560] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:24,903] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:24,907] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:25,009] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:52:25,289] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:25,624] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:25,627] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:52:25,731] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:52:26,039] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:52:26,374] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:52:39,118] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #6 rebalance_out_after_ops ============== [2022-03-01 03:52:39,178] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 03:52:40,002] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 03:52:40,007] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 03:52:40,007] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 03:52:40,021] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 03:52:40,029] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 03:52:40,036] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 03:52:40,044] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 03:52:40,049] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 03:52:40,051] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 03:52:40,072] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 03:52:40,073] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 03:52:40,079] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:52:40,098] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 03:52:40,100] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:52:40,159] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:52:40,161] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:52:50,171] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:53:00,186] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:53:20,215] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 03:53:20,215] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:53:30,254] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 03:53:30,265] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 03:53:30,266] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:30,273] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 03:53:30,278] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 03:53:30,278] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:30,282] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 03:53:30,285] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 03:53:30,285] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:30,288] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 03:53:30,291] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 03:53:30,291] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:30,294] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 03:53:30,297] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 03:53:30,297] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:30,300] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 03:53:30,303] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 03:53:30,303] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:30,306] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 03:53:30,306] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #6 rebalance_out_after_ops ============== [2022-03-01 03:53:30,306] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 03:53:30,307] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_6 ok ---------------------------------------------------------------------- Ran 1 test in 509.294s OK rebalance_out_with_queries (rebalance.rebalanceout.RebalanceOutTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_7 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_queries,nodes_out=1,blob_generator=False,value_size=1024,GROUP=OUT;BASIC;P0;FROM_2_0 Test Input params: {'nodes_out': '1', 'blob_generator': 'False', 'value_size': '1024', 'GROUP': 'OUT;BASIC;P0;FROM_2_0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 7, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_7'} [2022-03-01 03:53:30,330] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:30,474] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:53:30,754] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:31,134] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:31,144] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:53:31,168] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:53:31,168] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #7 rebalance_out_with_queries============== [2022-03-01 03:53:31,168] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:53:31,337] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:53:31,343] - [task:159] INFO - {'uptime': '2566', 'memoryTotal': 3947372544, 'memoryFree': 2804809728, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:53:31,351] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:53:31,351] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:53:31,351] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:53:31,362] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:53:31,362] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:53:31,362] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:53:31,377] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:53:31,388] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:53:31,388] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:53:31,392] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:53:31,392] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:53:31,393] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:53:31,393] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:53:31,513] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:53:31,515] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:31,656] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:53:31,929] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:32,312] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:32,313] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:53:32,391] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:53:32,391] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:53:32,395] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:32,398] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:32,408] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:53:32,438] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:53:32,442] - [task:159] INFO - {'uptime': '2565', 'memoryTotal': 3947372544, 'memoryFree': 3212247040, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:53:32,444] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:53:32,444] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 03:53:32,449] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:53:32,449] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:53:32,581] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:53:32,583] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:32,725] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:53:33,007] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:33,386] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:33,387] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:53:33,466] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:53:33,466] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:53:33,470] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:33,473] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:33,480] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:53:33,503] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:53:33,506] - [task:159] INFO - {'uptime': '2540', 'memoryTotal': 4201627648, 'memoryFree': 3550138368, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:53:33,507] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:53:33,508] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:53:33,511] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:53:33,511] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:53:33,570] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:53:33,572] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:33,676] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:53:33,985] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:34,316] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:34,316] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:53:34,384] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:53:34,384] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:53:34,387] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:34,390] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:34,394] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:53:34,407] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:53:34,409] - [task:159] INFO - {'uptime': '2541', 'memoryTotal': 4201627648, 'memoryFree': 3553415168, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:53:34,411] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:53:34,412] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:53:34,415] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:53:34,416] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:53:34,472] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:53:34,474] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:34,581] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:53:34,895] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:35,234] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:35,235] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:53:35,305] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:53:35,305] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:53:35,308] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:35,310] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:35,314] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:53:35,327] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:53:35,331] - [task:159] INFO - {'uptime': '2541', 'memoryTotal': 4201627648, 'memoryFree': 3516735488, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:53:35,333] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:53:35,333] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:53:35,337] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:53:35,337] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:53:35,400] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:53:35,402] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:35,505] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:53:35,790] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:36,120] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:36,121] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:53:36,191] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:53:36,191] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:53:36,194] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:36,197] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:36,200] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:53:36,216] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:53:36,219] - [task:159] INFO - {'uptime': '2545', 'memoryTotal': 4201627648, 'memoryFree': 3552301056, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:53:36,221] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:53:36,221] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:53:36,225] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:53:36,225] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:53:36,286] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:53:36,288] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:53:36,397] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:53:36,699] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:53:37,029] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:53:37,030] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:53:37,097] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:53:37,098] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:53:37,100] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:37,103] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:53:37,107] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:53:37,117] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:53:37,306] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:53:42,311] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:53:42,331] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:53:43,124] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:53:43,136] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:53:53,152] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:53:53,153] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:54:10,233] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 03:54:10,249] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:54:20,265] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:54:20,265] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:54:34,646] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 03:54:34,666] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:54:44,682] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:54:44,682] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:54:59,291] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 03:54:59,314] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:55:09,330] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:55:09,331] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:55:23,702] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 03:55:23,731] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:55:33,748] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:55:33,749] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:55:47,273] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 03:55:47,273] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 03:55:47,273] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 03:55:47,273] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 03:55:47,273] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 03:55:47,284] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:55:47,351] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:55:57,362] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:55:57,375] - [task:902] INFO - rebalancing was completed with progress: 100% in 10.023633480072021 sec [2022-03-01 03:55:57,377] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:55:57,513] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:55:57,792] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:55:58,169] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:55:58,170] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:55:58,249] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:55:58,250] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:55:58,250] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 03:55:58,409] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 03:55:58,444] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 03:55:58,444] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 03:55:59,299] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:55:59,450] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:55:59,577] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:55:59,757] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:55:59,879] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:00,024] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:00,398] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:56:00,578] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:56:00,760] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:56:00,926] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:56:01,058] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:01,249] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:02,153] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 03:56:02,155] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #7 rebalance_out_with_queries ============== [2022-03-01 03:56:02,169] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:02,306] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:56:02,581] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:02,963] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:02,966] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:03,101] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:56:03,389] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:03,776] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:03,780] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:03,886] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:56:04,219] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:04,559] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:04,562] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:04,669] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:56:04,986] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:05,325] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:05,328] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:05,432] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:56:05,726] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:06,067] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:06,069] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:06,174] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:56:06,482] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:06,817] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:19,018] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:56:19,018] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.20163766632549, 'mem_free': 2800418816, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:56:19,018] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 7.121212121212121, 'mem_free': 3377307648, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:56:19,018] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.852674066599395, 'mem_free': 3321544704, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 03:56:19,018] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.587281479604763, 'mem_free': 3346219008, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:56:19,018] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.991931417044881, 'mem_free': 3323781120, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 03:56:19,018] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 8.360780339498353, 'mem_free': 3032293376, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:56:19,019] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:56:19,019] - [basetestcase:559] INFO - Time to execute basesetup : 168.6904330253601 [2022-03-01 03:56:19,019] - [basetestcase:1445] INFO - create 1000 documents... [2022-03-01 03:56:19,102] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:56:19,216] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:56:19,344] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:56:19,427] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:56:19,510] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:19,647] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:20,140] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:56:20,142] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:56:20,147] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:56:20,148] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:56:20,372] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:56:20,378] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:56:20,379] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:56:20,381] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:56:20,464] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:56:20,465] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:56:20,466] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:56:20,467] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 03:56:20,552] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:56:20,554] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:56:20,555] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:56:20,556] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:56:20,698] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:20,700] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:20,701] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:20,701] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:56:20,802] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:20,808] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:20,810] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:20,810] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:56:21,098] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:56:21,098] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 03:56:21,367] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:21,506] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:56:21,789] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:22,172] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:22,173] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:56:22,648] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:56:22,656] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:22,756] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:56:23,056] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:23,384] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:23,384] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:56:23,628] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:56:23,635] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:23,736] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:56:24,008] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:24,345] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:24,346] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:56:24,595] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:56:24,602] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:24,705] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:56:25,024] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:25,371] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:25,371] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:56:25,623] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:56:25,630] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:25,736] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:56:26,046] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:26,376] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:26,377] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:56:26,618] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:56:26,626] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:56:26,770] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:56:27,042] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:56:27,433] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:56:27,434] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 03:56:27,901] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:56:28,019] - [rest_client:1073] ERROR - GET http://172.23.104.172:8092//default/_design/ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2022-03-01 03:56:28,020] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:56:28,066] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:56:28,214] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:56:28,393] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:56:28,603] - [rest_client:548] INFO - -->create_design_document [2022-03-01 03:56:28,833] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?stale=ok [2022-03-01 03:56:29,074] - [task:2865] INFO - view : default_view0 was created successfully in ddoc: ddoc1 [2022-03-01 03:56:29,079] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?stale=ok [2022-03-01 03:56:29,087] - [task:2865] INFO - view : default_view1 was created successfully in ddoc: ddoc1 [2022-03-01 03:56:29,091] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?stale=ok [2022-03-01 03:56:29,100] - [task:2865] INFO - view : default_view2 was created successfully in ddoc: ddoc1 [2022-03-01 03:56:29,103] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?stale=ok [2022-03-01 03:56:29,113] - [task:2865] INFO - view : default_view3 was created successfully in ddoc: ddoc1 [2022-03-01 03:56:29,117] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?stale=ok [2022-03-01 03:56:29,126] - [task:2865] INFO - view : default_view4 was created successfully in ddoc: ddoc1 [2022-03-01 03:56:30,140] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true [2022-03-01 03:56:31,161] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true [2022-03-01 03:56:32,239] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true [2022-03-01 03:56:33,323] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true [2022-03-01 03:56:34,404] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true [2022-03-01 03:56:35,475] - [task:3740] INFO - task for monitoring indexer:_design/ddoc1 completed [2022-03-01 03:56:35,479] - [task:3740] INFO - task for monitoring indexer:_design/ddoc1 completed [2022-03-01 03:56:35,482] - [task:3740] INFO - task for monitoring indexer:_design/ddoc1 completed [2022-03-01 03:56:35,486] - [task:3740] INFO - task for monitoring indexer:_design/ddoc1 completed [2022-03-01 03:56:35,489] - [task:3740] INFO - task for monitoring indexer:_design/ddoc1 completed [2022-03-01 03:56:35,492] - [task:3740] INFO - task for monitoring indexer:_design/ddoc1 completed [2022-03-01 03:56:36,502] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:36,577] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:36,654] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:36,730] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:36,808] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:36,884] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:36,952] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view0, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:56:36,952] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:56:36,960] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:37,020] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view1, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:56:37,020] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:56:37,029] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:37,089] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view2, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:56:37,089] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:56:37,098] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:37,167] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view3, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:56:37,167] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:56:37,177] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:37,257] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view4, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:56:37,258] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:56:37,269] - [basetestcase:829] INFO - sleep for 12 secs. ... [2022-03-01 03:56:38,315] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.162', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:56:38,372] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:56:48,388] - [rest_client:2012] INFO - rebalance percentage : 18.00 % [2022-03-01 03:56:48,388] - [task:843] INFO - Rebalance - status: running, progress: 18.00% [2022-03-01 03:56:49,428] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:59,101] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:59,178] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:59,256] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:59,330] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:59,394] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:56:59,465] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view0, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:56:59,466] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:09,484] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:57:09,484] - [task:843] INFO - Rebalance - status: running, progress: 50.00% [2022-03-01 03:57:09,511] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:09,587] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view1, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:09,587] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:09,594] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:09,672] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view2, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:09,672] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:09,680] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:09,751] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view3, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:09,751] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:09,759] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:09,824] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view4, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:09,824] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:29,879] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:57:29,879] - [task:843] INFO - Rebalance - status: running, progress: 50.00% [2022-03-01 03:57:49,931] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 03:57:49,955] - [task:902] INFO - rebalancing was completed with progress: 100% in 71.58189916610718 sec [2022-03-01 03:57:50,966] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,045] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,119] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,195] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,315] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,392] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,459] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view0, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:51,459] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:51,467] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,546] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view1, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:51,546] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:51,556] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,623] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view2, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:51,623] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:51,631] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,696] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view3, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:51,696] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:51,705] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 03:57:51,777] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: ddoc1, View: default_view4, (1000 rows) expected, (1000 rows) returned [2022-03-01 03:57:51,777] - [task:3084] INFO - expected number of rows: '1000' was found for view query [2022-03-01 03:57:51,800] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:57:51,812] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:57:51,822] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:57:51,835] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:57:52,780] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:57:52,914] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:57:53,184] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:57:53,577] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:57:53,578] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:57:53,879] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:57:53,880] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 03:57:53,882] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:57:54,019] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:57:54,294] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:57:54,682] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:57:54,683] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:57:54,979] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:57:54,980] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 03:57:54,982] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:57:55,088] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:57:55,398] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:57:55,726] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:57:55,727] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:57:55,896] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:57:55,898] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 03:57:55,900] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:57:56,002] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:57:56,304] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:57:56,640] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:57:56,641] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:57:56,812] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:57:56,813] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 03:57:56,816] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:57:56,921] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:57:57,207] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:57:57,543] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:57:57,543] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:57:57,711] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:57:57,712] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 03:57:57,801] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 03:57:57,902] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 03:57:58,032] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 03:57:58,109] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 03:57:58,187] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 03:57:58,364] - [task:2319] INFO - 1000 items will be verified on default bucket [2022-03-01 03:57:58,714] - [task:2328] INFO - 0 items were verified [2022-03-01 03:57:58,900] - [task:2332] INFO - 1000 items were verified in 0.5349128246307373 sec.the average number of ops - 1869.0 per second [2022-03-01 03:57:58,924] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 03:57:58,935] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 03:57:59,728] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:57:59,871] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:58:00,147] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:00,542] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:00,544] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:00,843] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:00,846] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:00,981] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:58:01,263] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:01,656] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:01,657] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:01,952] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:01,955] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:02,059] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:58:02,364] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:02,700] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:02,700] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:02,873] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:02,876] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:02,980] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:58:03,289] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:03,628] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:03,629] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:03,803] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:03,806] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:03,911] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:58:04,198] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:04,540] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:04,542] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:04,716] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:04,717] - [task:972] INFO - Saw curr_items 1000 == 1000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:58:04,719] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:04,856] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:58:05,120] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:05,497] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:05,497] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:05,774] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:05,778] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:05,917] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:58:06,178] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:06,560] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:06,561] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:06,819] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:06,822] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:06,923] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:58:07,223] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:07,545] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:07,546] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:07,718] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:07,721] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:07,823] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:58:08,121] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:08,463] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:08,463] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:08,645] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:08,648] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:08,754] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:58:09,044] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:09,388] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:09,389] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:09,564] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:09,565] - [task:972] INFO - Saw vb_active_curr_items 1000 == 1000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:58:09,567] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:09,705] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:58:09,976] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:10,355] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:10,356] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:10,647] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:10,651] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:10,787] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:58:11,055] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:11,457] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:11,458] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:11,759] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:11,762] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:11,863] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:58:12,167] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:12,496] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:12,497] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:12,668] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:12,671] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:12,774] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:58:13,081] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:13,415] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:13,416] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:13,584] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:13,587] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:13,690] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:58:13,975] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:14,311] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:14,311] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:14,484] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:14,485] - [task:972] INFO - Saw vb_replica_curr_items 1000 == 1000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:58:14,487] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:14,622] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:58:14,902] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:15,283] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:15,284] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:15,579] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:15,582] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:15,716] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:58:15,991] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:16,368] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:16,369] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:16,645] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:16,648] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:16,753] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:58:17,059] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:17,390] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:17,391] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:17,559] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:17,562] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:17,665] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:58:17,968] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:18,303] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:18,304] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:18,473] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:18,476] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:18,580] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:58:18,856] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:19,189] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:19,190] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 03:58:19,363] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:58:19,365] - [task:972] INFO - Saw curr_items_tot 2000 == 2000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 03:58:19,365] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 03:58:20,718] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 188 [2022-03-01 03:58:20,718] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 03:58:20,718] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 205 [2022-03-01 03:58:20,718] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 205 [2022-03-01 03:58:20,718] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 202 [2022-03-01 03:58:20,718] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 03:58:20,718] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 205 [2022-03-01 03:58:20,719] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 205 [2022-03-01 03:58:20,719] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 192 [2022-03-01 03:58:20,719] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 03:58:20,719] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 03:58:20,719] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 03:58:20,719] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 217 [2022-03-01 03:58:20,719] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 03:58:20,719] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 03:58:20,719] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 03:58:20,719] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 201 [2022-03-01 03:58:20,720] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 03:58:20,720] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 03:58:20,720] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 03:58:20,720] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 03:58:20,720] - [rebalance_helper:390] INFO - sum : 1000 and sum * (replica_factor + 1) (2) : 2000 [2022-03-01 03:58:21,164] - [rebalance_helper:393] INFO - curr_items_tot from master: 2000 [2022-03-01 03:58:21,165] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 03:58:21,187] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 03:58:21,187] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.375412750825502, 'mem_free': 2764488704, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 03:58:21,188] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.707440100882724, 'mem_free': 3302502400, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 03:58:21,188] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 4.16141235813367, 'mem_free': 3264573440, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 03:58:21,188] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.655155028989161, 'mem_free': 3290664960, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 03:58:21,188] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.462240243284339, 'mem_free': 2964398080, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 03:58:21,188] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 03:58:21,190] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:21,324] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:58:21,593] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:21,987] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:21,989] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:22,127] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:58:22,407] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:22,794] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:22,798] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:22,900] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:58:23,209] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:23,547] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:23,550] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:23,651] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:58:23,951] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:24,291] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:24,293] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:24,395] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:58:24,684] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:25,016] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:25,020] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:58:25,128] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:58:25,438] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:58:25,764] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:58:38,224] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #7 rebalance_out_with_queries ============== [2022-03-01 03:58:38,290] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 03:58:39,075] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 03:58:39,081] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 03:58:39,082] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 03:58:39,093] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 03:58:39,102] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 03:58:39,110] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 03:58:39,118] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 03:58:39,123] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 03:58:39,125] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 03:58:39,150] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 03:58:39,151] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 03:58:39,159] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:58:39,184] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 03:58:39,187] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 03:58:39,244] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 03:58:39,246] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 03:58:49,261] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:58:59,276] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 03:59:19,305] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 03:59:19,306] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 03:59:29,355] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 03:59:29,362] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 03:59:29,362] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:29,367] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 03:59:29,371] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 03:59:29,371] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:29,375] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 03:59:29,378] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 03:59:29,379] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:29,382] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 03:59:29,385] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 03:59:29,385] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:29,388] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 03:59:29,391] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 03:59:29,391] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:29,394] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 03:59:29,398] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 03:59:29,398] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:29,400] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 03:59:29,401] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #7 rebalance_out_with_queries ============== [2022-03-01 03:59:29,401] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 03:59:29,401] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_7 ok ---------------------------------------------------------------------- Ran 1 test in 359.082s OK incremental_rebalance_out_with_queries (rebalance.rebalanceout.RebalanceOutTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_8 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t rebalance.rebalanceout.RebalanceOutTests.incremental_rebalance_out_with_queries,blob_generator=False,items=10000,GROUP=OUT;P0;FROM_2_0 Test Input params: {'blob_generator': 'False', 'items': '10000', 'GROUP': 'OUT;P0;FROM_2_0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_8'} [2022-03-01 03:59:29,421] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:29,559] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:59:29,840] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:30,219] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:30,227] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 03:59:30,242] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 03:59:30,242] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #8 incremental_rebalance_out_with_queries============== [2022-03-01 03:59:30,243] - [basetestcase:331] INFO - initializing cluster [2022-03-01 03:59:30,427] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:59:30,432] - [task:159] INFO - {'uptime': '2925', 'memoryTotal': 3947372544, 'memoryFree': 2761162752, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:59:30,440] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 03:59:30,440] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 03:59:30,440] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 03:59:30,455] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 03:59:30,455] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 03:59:30,455] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 03:59:30,471] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 03:59:30,481] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 03:59:30,481] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 03:59:30,484] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 03:59:30,484] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 03:59:30,484] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:59:30,485] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 03:59:30,629] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:59:30,631] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:30,772] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 03:59:31,052] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:31,430] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:31,431] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:59:31,474] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:59:31,474] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:59:31,478] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:31,480] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:31,488] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:59:31,526] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:59:31,530] - [task:159] INFO - {'uptime': '2923', 'memoryTotal': 3947372544, 'memoryFree': 3202383872, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 03:59:31,532] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:59:31,533] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 03:59:31,537] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:59:31,537] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 03:59:31,668] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:59:31,670] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:31,811] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 03:59:32,082] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:32,484] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:32,484] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:59:32,565] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:59:32,565] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:59:32,569] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:32,572] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:32,578] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:59:32,602] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:59:32,605] - [task:159] INFO - {'uptime': '2899', 'memoryTotal': 4201627648, 'memoryFree': 3559321600, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:59:32,607] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:59:32,608] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:59:32,611] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:59:32,611] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 03:59:32,670] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:59:32,674] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:32,778] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 03:59:33,100] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:33,433] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:33,434] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:59:33,502] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:59:33,502] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:59:33,506] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:33,508] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:33,512] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:59:33,527] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:59:33,530] - [task:159] INFO - {'uptime': '2900', 'memoryTotal': 4201627648, 'memoryFree': 3562246144, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:59:33,532] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:59:33,532] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:59:33,536] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:59:33,536] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 03:59:33,594] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:59:33,597] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:33,700] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 03:59:34,002] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:34,340] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:34,340] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:59:34,409] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:59:34,409] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:59:34,412] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:34,415] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:34,419] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:59:34,433] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:59:34,436] - [task:159] INFO - {'uptime': '2900', 'memoryTotal': 4201627648, 'memoryFree': 3516735488, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:59:34,437] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:59:34,438] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:59:34,441] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:59:34,441] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 03:59:34,504] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:59:34,506] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:34,612] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 03:59:34,897] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:35,231] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:35,232] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:59:35,301] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:59:35,301] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:59:35,304] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:35,307] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:35,311] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:59:35,325] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 03:59:35,328] - [task:159] INFO - {'uptime': '2903', 'memoryTotal': 4201627648, 'memoryFree': 3543695360, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 03:59:35,330] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 03:59:35,330] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 03:59:35,334] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 03:59:35,334] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 03:59:35,398] - [rest_client:1151] INFO - --> status:True [2022-03-01 03:59:35,400] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 03:59:35,504] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 03:59:35,816] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 03:59:36,150] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 03:59:36,151] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 03:59:36,222] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 03:59:36,222] - [remote_util:5219] INFO - ['ok'] [2022-03-01 03:59:36,225] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:36,227] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 03:59:36,231] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 03:59:36,244] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 03:59:36,409] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 03:59:41,415] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 03:59:41,434] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 03:59:42,251] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 03:59:42,261] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 03:59:52,278] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 03:59:52,278] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:00:08,709] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 04:00:08,726] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:00:18,744] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:00:18,744] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:00:32,663] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 04:00:32,681] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:00:42,699] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:00:42,699] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:00:57,184] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 04:00:57,202] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:01:07,221] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:01:07,221] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:01:21,721] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 04:01:21,741] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:01:31,759] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:01:31,759] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:01:46,012] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 04:01:46,013] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 04:01:46,013] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 04:01:46,013] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 04:01:46,013] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 04:01:46,028] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:01:46,092] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:01:56,106] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:01:56,120] - [task:902] INFO - rebalancing was completed with progress: 100% in 10.027526617050171 sec [2022-03-01 04:01:56,123] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:01:56,264] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:01:56,537] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:01:56,920] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:01:56,921] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:01:56,999] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:01:56,999] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:01:57,000] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 04:01:57,153] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 04:01:57,187] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 04:01:57,187] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 04:01:58,049] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:01:58,205] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:01:58,340] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:01:58,512] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:01:58,656] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:01:58,813] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:01:59,164] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:01:59,350] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:01:59,532] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:01:59,652] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:01:59,754] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:01:59,918] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:02:00,730] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 04:02:00,732] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #8 incremental_rebalance_out_with_queries ============== [2022-03-01 04:02:00,745] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:00,888] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:02:01,163] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:01,552] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:01,555] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:01,695] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:02:01,980] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:02,374] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:02,377] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:02,485] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:02:02,808] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:03,144] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:03,146] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:03,258] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:02:03,568] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:03,906] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:03,909] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:04,017] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:02:04,328] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:04,666] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:04,670] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:04,776] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:02:05,097] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:05,431] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:17,701] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:02:17,701] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 12.38192494255808, 'mem_free': 2769375232, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:02:17,701] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.188431422076282, 'mem_free': 3358973952, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 04:02:17,702] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 10.06066734074823, 'mem_free': 3310960640, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 04:02:17,702] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.682219419924338, 'mem_free': 3350212608, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 04:02:17,702] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.917255297679112, 'mem_free': 3320594432, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 04:02:17,702] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.13545407901488, 'mem_free': 3024261120, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:02:17,702] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:02:17,702] - [basetestcase:559] INFO - Time to execute basesetup : 168.28311705589294 [2022-03-01 04:02:17,702] - [basetestcase:1445] INFO - create 10000 documents... [2022-03-01 04:02:17,782] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:02:17,900] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:02:18,035] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:02:18,122] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:02:18,205] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:02:18,344] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:02:18,812] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:02:18,817] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:02:18,820] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:02:18,822] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:02:19,032] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:02:19,037] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:02:19,047] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:02:19,051] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:02:19,123] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:02:19,129] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:02:19,129] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:02:19,136] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:02:19,209] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:02:19,211] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:02:19,216] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:02:19,219] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:02:19,369] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:02:19,371] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:02:19,371] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:02:19,374] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:02:19,473] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:02:19,473] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:02:19,475] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:02:19,478] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:02:20,073] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:02:20,074] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:02:20,331] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:20,465] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:02:20,728] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:21,106] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:21,106] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:02:21,582] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:02:21,591] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:21,697] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:02:22,016] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:22,364] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:22,365] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:02:22,619] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:02:22,626] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:22,733] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:02:23,016] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:23,353] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:23,354] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:02:23,607] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:02:23,615] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:23,718] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:02:24,031] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:24,368] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:24,369] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:02:24,613] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:02:24,621] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:24,724] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:02:25,035] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:25,366] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:25,367] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:02:25,609] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:02:25,617] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:02:25,752] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:02:26,024] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:02:26,407] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:02:26,407] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:02:26,878] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:02:27,708] - [rest_client:1073] ERROR - GET http://172.23.104.172:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2022-03-01 04:02:27,710] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:02:27,749] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:02:27,895] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:02:28,069] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:02:28,266] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:02:28,494] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?stale=ok [2022-03-01 04:02:33,745] - [task:2865] INFO - view : default_view0 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:02:33,749] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?stale=ok [2022-03-01 04:02:33,772] - [task:2865] INFO - view : default_view1 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:02:33,776] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?stale=ok [2022-03-01 04:02:33,799] - [task:2865] INFO - view : default_view2 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:02:33,802] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?stale=ok [2022-03-01 04:02:33,825] - [task:2865] INFO - view : default_view3 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:02:33,829] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?stale=ok [2022-03-01 04:02:33,851] - [task:2865] INFO - view : default_view4 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:02:34,867] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true [2022-03-01 04:02:35,887] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true [2022-03-01 04:02:37,438] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true [2022-03-01 04:02:39,036] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true [2022-03-01 04:02:40,601] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true [2022-03-01 04:02:42,141] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:42,146] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:42,152] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:42,158] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:42,169] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:42,184] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:42,184] - [basetestcase:829] INFO - sleep for 2 secs. ... [2022-03-01 04:02:45,194] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:45,199] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:45,202] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:45,212] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:45,225] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:45,236] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:45,236] - [basetestcase:829] INFO - sleep for 2 secs. ... [2022-03-01 04:02:48,244] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:48,249] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:48,252] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:48,256] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:48,259] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:48,262] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:02:48,263] - [basetestcase:829] INFO - sleep for 2 secs. ... [2022-03-01 04:02:51,276] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:51,848] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:52,418] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:53,008] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:53,524] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:54,099] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:54,623] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:02:54,623] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:02:54,635] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:55,137] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:02:55,137] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:02:55,148] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:55,741] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:02:55,741] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:02:55,752] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:56,305] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:02:56,305] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:02:56,316] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:02:56,848] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:02:56,848] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:02:56,863] - [basetestcase:829] INFO - sleep for 12 secs. ... [2022-03-01 04:02:57,906] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.162', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:02:57,967] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:03:07,982] - [rest_client:2012] INFO - rebalance percentage : 18.00 % [2022-03-01 04:03:07,982] - [task:843] INFO - Rebalance - status: running, progress: 18.00% [2022-03-01 04:03:09,042] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:09,784] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:10,543] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:11,252] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:11,918] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:16,953] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:17,663] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (9994 rows) returned [2022-03-01 04:03:17,674] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:18,448] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (9986 rows) returned [2022-03-01 04:03:18,462] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:19,150] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9957 rows) returned [2022-03-01 04:03:19,161] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:24,512] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (9993 rows) returned [2022-03-01 04:03:24,523] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:25,045] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9982 rows) returned [2022-03-01 04:03:35,063] - [rest_client:2012] INFO - rebalance percentage : 25.00 % [2022-03-01 04:03:35,063] - [task:843] INFO - Rebalance - status: running, progress: 25.00% [2022-03-01 04:03:35,091] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:35,658] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:03:35,658] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:03:35,670] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:36,232] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:03:36,232] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:03:36,245] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:36,832] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:03:36,833] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:03:36,845] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:37,399] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:03:37,400] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:03:37,411] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:03:37,975] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:03:37,975] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:03:56,001] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:03:56,002] - [task:843] INFO - Rebalance - status: running, progress: 50.00% [2022-03-01 04:04:16,042] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:04:16,063] - [task:902] INFO - rebalancing was completed with progress: 100% in 78.09590721130371 sec [2022-03-01 04:04:17,073] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:17,616] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:18,212] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:18,760] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:19,331] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:19,879] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:20,476] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:04:20,476] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:04:20,488] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:21,019] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:04:21,019] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:04:21,030] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:21,558] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:04:21,559] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:04:21,568] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:22,103] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:04:22,104] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:04:22,114] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:04:22,652] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:04:22,653] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:04:22,678] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:04:22,689] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:04:22,700] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:04:22,709] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:04:23,659] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:23,795] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:04:24,074] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:24,473] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:24,474] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:24,741] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:24,742] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:04:24,744] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:24,884] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:04:25,157] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:25,540] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:25,541] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:25,794] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:25,795] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:04:25,797] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:25,906] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:04:26,219] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:26,551] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:26,552] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:26,729] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:26,730] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:04:26,733] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:26,837] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:04:27,147] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:27,489] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:27,490] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:27,662] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:27,663] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:04:27,665] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:27,768] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:04:28,057] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:28,392] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:28,393] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:28,566] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:28,567] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:04:28,712] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:04:28,806] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:04:28,879] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:04:28,999] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:04:29,080] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:04:29,223] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:04:29,569] - [task:2328] INFO - 0 items were verified [2022-03-01 04:04:31,321] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:04:31,322] - [task:2332] INFO - 10000 items were verified in 2.0979866981506348 sec.the average number of ops - 4766.0 per second [2022-03-01 04:04:31,345] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:04:31,357] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:04:31,584] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:31,721] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:04:31,995] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:32,379] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:32,380] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:32,671] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:32,674] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:32,810] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:04:33,082] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:33,465] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:33,466] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:33,755] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:33,758] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:33,865] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:04:34,175] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:34,508] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:34,509] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:34,682] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:34,685] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:34,789] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:04:35,094] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:35,436] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:35,436] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:35,611] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:35,614] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:35,713] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:04:35,987] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:36,315] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:36,315] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:36,489] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:36,490] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 04:04:36,492] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:36,627] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:04:36,893] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:37,278] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:37,279] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:37,543] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:37,546] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:37,680] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:04:37,952] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:38,333] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:38,334] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:38,585] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:38,588] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:38,690] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:04:38,992] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:39,324] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:39,324] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:39,495] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:39,498] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:39,597] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:04:39,897] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:40,230] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:40,231] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:40,407] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:40,410] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:40,515] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:04:40,796] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:41,127] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:41,128] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:41,300] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:41,302] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 04:04:41,304] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:41,447] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:04:41,754] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:42,145] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:42,146] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:42,463] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:42,465] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:42,599] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:04:42,861] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:43,245] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:43,246] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:43,542] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:43,545] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:43,652] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:04:43,959] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:44,296] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:44,297] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:44,469] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:44,472] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:44,574] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:04:44,876] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:45,210] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:45,211] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:45,392] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:45,396] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:45,500] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:04:45,786] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:46,121] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:46,121] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:46,294] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:46,295] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 04:04:46,297] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:46,433] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:04:46,702] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:47,077] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:47,078] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:47,407] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:47,410] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:47,550] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:04:47,841] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:48,222] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:48,223] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:48,521] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:48,524] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:48,630] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:04:48,943] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:49,276] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:49,277] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:49,448] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:49,452] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:49,553] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:04:49,865] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:50,202] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:50,203] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:50,377] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:50,380] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:04:50,482] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:04:50,765] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:04:51,098] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:04:51,099] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:04:51,268] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:04:51,270] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091',default bucket [2022-03-01 04:04:51,270] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:04:52,696] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 2044 [2022-03-01 04:04:52,696] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 04:04:52,696] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 205 [2022-03-01 04:04:52,696] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 205 [2022-03-01 04:04:52,697] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1978 [2022-03-01 04:04:52,697] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 04:04:52,697] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 205 [2022-03-01 04:04:52,697] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 205 [2022-03-01 04:04:52,697] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 1913 [2022-03-01 04:04:52,697] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 04:04:52,697] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 04:04:52,697] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 04:04:52,697] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 2035 [2022-03-01 04:04:52,697] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 04:04:52,698] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 04:04:52,698] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 04:04:52,698] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 2030 [2022-03-01 04:04:52,698] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 04:04:52,698] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 04:04:52,698] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 04:04:52,698] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 04:04:52,698] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:04:53,143] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 04:04:53,143] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:04:53,154] - [basetestcase:829] INFO - sleep for 12 secs. ... [2022-03-01 04:04:53,327] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:04:53,392] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:05:03,408] - [rest_client:2012] INFO - rebalance percentage : 5.00 % [2022-03-01 04:05:03,408] - [task:843] INFO - Rebalance - status: running, progress: 5.00% [2022-03-01 04:05:05,448] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:06,128] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:11,689] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:12,560] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:13,327] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:14,163] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:15,025] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (9986 rows) returned [2022-03-01 04:05:15,036] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:15,894] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (9997 rows) returned [2022-03-01 04:05:15,912] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:16,689] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9987 rows) returned [2022-03-01 04:05:16,700] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:17,455] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (9951 rows) returned [2022-03-01 04:05:17,467] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:18,306] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9991 rows) returned [2022-03-01 04:05:28,323] - [rest_client:2012] INFO - rebalance percentage : 19.00 % [2022-03-01 04:05:28,323] - [task:843] INFO - Rebalance - status: running, progress: 19.00% [2022-03-01 04:05:28,358] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:29,016] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (8428 rows) returned [2022-03-01 04:05:29,030] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:29,867] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:05:29,867] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:05:29,878] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:30,644] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9990 rows) returned [2022-03-01 04:05:30,671] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:31,455] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (9986 rows) returned [2022-03-01 04:05:31,467] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:32,279] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9951 rows) returned [2022-03-01 04:05:32,312] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:33,108] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (9992 rows) returned [2022-03-01 04:05:33,122] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:33,926] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9944 rows) returned [2022-03-01 04:05:33,938] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:34,643] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (9971 rows) returned [2022-03-01 04:05:34,654] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:35,462] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9997 rows) returned [2022-03-01 04:05:35,473] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:36,194] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (9987 rows) returned [2022-03-01 04:05:36,205] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:36,898] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9971 rows) returned [2022-03-01 04:05:36,912] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:37,721] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:05:37,722] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:05:37,733] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:38,513] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9995 rows) returned [2022-03-01 04:05:48,530] - [rest_client:2012] INFO - rebalance percentage : 25.00 % [2022-03-01 04:05:48,530] - [task:843] INFO - Rebalance - status: running, progress: 25.00% [2022-03-01 04:05:48,558] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:49,132] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (9852 rows) returned [2022-03-01 04:05:49,144] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:49,634] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:05:49,635] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:05:49,645] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:50,159] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:05:50,159] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:05:51,172] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:05:51,722] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:05:51,722] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:06:08,747] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:06:08,748] - [task:843] INFO - Rebalance - status: running, progress: 50.00% [2022-03-01 04:06:28,787] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:06:28,815] - [task:902] INFO - rebalancing was completed with progress: 100% in 95.4225811958313 sec [2022-03-01 04:06:29,825] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:30,328] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:30,887] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:31,383] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:31,887] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:32,439] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:32,955] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:06:32,956] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:06:32,965] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:33,507] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:06:33,507] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:06:33,517] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:33,990] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:06:33,991] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:06:34,001] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:34,504] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:06:34,504] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:06:34,515] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:06:35,003] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:06:35,003] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:06:35,026] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:06:35,036] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:06:35,045] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:06:35,056] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:06:36,009] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:36,149] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:06:36,421] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:36,801] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:36,802] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:37,104] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:37,105] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:06:37,107] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:37,245] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:06:37,553] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:37,945] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:37,946] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:38,249] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:38,250] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:06:38,252] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:38,355] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:06:38,666] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:38,993] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:38,994] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:39,122] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:39,124] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:06:39,250] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:06:39,339] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:06:39,414] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:06:39,600] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:06:40,126] - [task:2328] INFO - 0 items were verified [2022-03-01 04:06:41,515] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:06:41,515] - [task:2332] INFO - 10000 items were verified in 1.9145326614379883 sec.the average number of ops - 5223.0 per second [2022-03-01 04:06:41,536] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:06:41,546] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:06:42,140] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:42,275] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:06:42,552] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:42,943] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:42,945] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:43,243] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:43,246] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:43,387] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:06:43,659] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:44,038] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:44,039] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:44,312] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:44,315] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:44,414] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:06:44,713] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:45,048] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:45,049] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:45,220] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:45,221] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 04:06:45,223] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:45,360] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:06:45,623] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:46,001] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:46,002] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:46,297] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:46,300] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:46,432] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:06:46,703] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:47,085] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:47,086] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:47,396] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:47,399] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:47,505] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:06:47,806] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:48,133] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:48,134] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:48,302] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:48,303] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 04:06:48,305] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:48,445] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:06:48,720] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:49,100] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:49,101] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:49,402] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:49,405] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:49,542] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:06:49,804] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:50,188] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:50,189] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:50,478] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:50,481] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:50,583] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:06:50,895] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:51,223] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:51,224] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:51,396] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:51,397] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 04:06:51,399] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:51,532] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:06:51,804] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:52,188] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:52,189] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:52,449] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:52,452] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:52,591] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:06:52,863] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:53,248] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:53,249] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:53,549] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:53,552] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:06:53,656] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:06:53,972] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:06:54,296] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:06:54,297] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:06:54,467] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:06:54,468] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091',default bucket [2022-03-01 04:06:54,469] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:06:55,444] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 3309 [2022-03-01 04:06:55,444] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 04:06:55,445] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 342 [2022-03-01 04:06:55,445] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 342 [2022-03-01 04:06:55,445] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 3387 [2022-03-01 04:06:55,445] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 04:06:55,445] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 341 [2022-03-01 04:06:55,445] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 341 [2022-03-01 04:06:55,445] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 3304 [2022-03-01 04:06:55,445] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 04:06:55,445] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 341 [2022-03-01 04:06:55,445] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 341 [2022-03-01 04:06:55,446] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 04:06:55,446] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:06:55,857] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 04:06:55,857] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:06:55,867] - [basetestcase:829] INFO - sleep for 12 secs. ... [2022-03-01 04:06:56,511] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.106.52,ns_1@172.23.105.158', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:06:56,566] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:07:06,581] - [rest_client:2012] INFO - rebalance percentage : 2.00 % [2022-03-01 04:07:06,582] - [task:843] INFO - Rebalance - status: running, progress: 2.00% [2022-03-01 04:07:08,619] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:09,354] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:09,964] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:10,710] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:11,457] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:12,116] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:12,835] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:07:12,835] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:07:12,845] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:13,576] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (9988 rows) returned [2022-03-01 04:07:13,597] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:14,295] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9987 rows) returned [2022-03-01 04:07:14,305] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:15,020] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:07:15,020] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:07:15,030] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:15,704] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9988 rows) returned [2022-03-01 04:07:15,715] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:16,450] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:07:16,451] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:07:16,470] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:17,226] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (9988 rows) returned [2022-03-01 04:07:27,245] - [rest_client:2012] INFO - rebalance percentage : 7.00 % [2022-03-01 04:07:27,245] - [task:843] INFO - Rebalance - status: running, progress: 7.00% [2022-03-01 04:07:27,275] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:28,081] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9072 rows) returned [2022-03-01 04:07:28,091] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:28,921] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:07:28,921] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:07:30,933] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:31,905] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9741 rows) returned [2022-03-01 04:07:33,919] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:34,714] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9855 rows) returned [2022-03-01 04:07:36,728] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:37,466] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9757 rows) returned [2022-03-01 04:07:47,482] - [rest_client:2012] INFO - rebalance percentage : 13.00 % [2022-03-01 04:07:47,483] - [task:843] INFO - Rebalance - status: running, progress: 13.00% [2022-03-01 04:07:47,511] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:48,282] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (8994 rows) returned [2022-03-01 04:07:50,301] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:51,215] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9832 rows) returned [2022-03-01 04:07:53,235] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:54,110] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9787 rows) returned [2022-03-01 04:07:56,124] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:07:56,996] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9751 rows) returned [2022-03-01 04:08:08,014] - [rest_client:2012] INFO - rebalance percentage : 20.00 % [2022-03-01 04:08:08,014] - [task:843] INFO - Rebalance - status: running, progress: 20.00% [2022-03-01 04:08:08,039] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:08:08,600] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (9314 rows) returned [2022-03-01 04:08:10,612] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:08:11,258] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:08:11,259] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:08:28,284] - [rest_client:2012] INFO - rebalance percentage : 25.00 % [2022-03-01 04:08:28,284] - [task:843] INFO - Rebalance - status: running, progress: 25.00% [2022-03-01 04:08:48,329] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:08:48,329] - [task:843] INFO - Rebalance - status: running, progress: 50.00% [2022-03-01 04:09:08,366] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:09:08,395] - [task:902] INFO - rebalancing was completed with progress: 100% in 131.82834696769714 sec [2022-03-01 04:09:09,404] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:09,642] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:09,786] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:09,925] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:10,074] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:10,278] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:10,407] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:09:10,408] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:09:10,417] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:10,548] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:09:10,549] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:09:10,559] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:10,695] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:09:10,695] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:09:10,706] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:10,843] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:09:10,843] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:09:10,853] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after [2022-03-01 04:09:11,048] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:09:11,048] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:09:11,069] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise'] [2022-03-01 04:09:11,078] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 04:09:11,086] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise'] [2022-03-01 04:09:11,095] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 04:09:12,054] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:12,193] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:12,467] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:12,842] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:12,843] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:09:13,133] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:13,134] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:09:13,198] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:09:13,306] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:09:14,136] - [task:2328] INFO - 0 items were verified [2022-03-01 04:09:14,895] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:09:14,895] - [task:2332] INFO - 10000 items were verified in 1.5888938903808594 sec.the average number of ops - 6293.0 per second [2022-03-01 04:09:14,915] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise'] [2022-03-01 04:09:14,924] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot] [2022-03-01 04:09:15,150] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:15,287] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:15,583] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:15,966] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:15,966] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:09:16,222] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:16,223] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:09:16,225] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:16,364] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:16,629] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:17,006] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:17,007] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:09:17,301] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:17,302] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:09:17,304] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:17,451] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:17,716] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:18,093] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:18,094] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:09:18,389] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:18,390] - [task:972] INFO - Saw vb_replica_curr_items 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:09:18,392] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:18,529] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:18,804] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:19,190] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:19,191] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:09:19,482] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:19,484] - [task:972] INFO - Saw curr_items_tot 10000 == 10000 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:09:19,484] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:09:19,904] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 10000 [2022-03-01 04:09:19,904] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 04:09:19,904] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 1024 [2022-03-01 04:09:19,904] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 0 [2022-03-01 04:09:19,905] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 0 [2022-03-01 04:09:19,905] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:09:20,273] - [rebalance_helper:393] INFO - curr_items_tot from master: 10000 [2022-03-01 04:09:20,274] - [rebalance_helper:397] WARNING - the number of nodes is less than replica requires [2022-03-01 04:09:20,274] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:09:20,286] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:09:20,286] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.88981975120589, 'mem_free': 2609422336, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:09:20,286] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:09:20,288] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:20,426] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:20,699] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:21,095] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:21,098] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:21,238] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:09:21,519] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:21,902] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:21,905] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:22,009] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:09:22,321] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:22,652] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:22,655] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:22,760] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:09:23,074] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:23,409] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:23,412] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:23,516] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:09:23,800] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:24,132] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:24,136] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:24,241] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:09:24,556] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:24,882] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:37,244] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #8 incremental_rebalance_out_with_queries ============== [2022-03-01 04:09:37,286] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 04:09:38,368] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 04:09:38,373] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 04:09:38,374] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 04:09:38,382] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 04:09:38,388] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:09:38,393] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:09:38,399] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:09:38,404] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:09:38,405] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 04:09:38,421] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 04:09:38,422] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 04:09:38,427] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,450] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 04:09:38,451] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,456] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 04:09:38,460] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 04:09:38,460] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,464] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 04:09:38,467] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 04:09:38,467] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,470] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 04:09:38,473] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 04:09:38,473] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,476] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 04:09:38,479] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 04:09:38,480] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,482] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 04:09:38,486] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 04:09:38,486] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:38,489] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 04:09:38,489] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #8 incremental_rebalance_out_with_queries ============== [2022-03-01 04:09:38,489] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 04:09:38,489] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_8 ok ---------------------------------------------------------------------- Ran 1 test in 609.079s OK pytests/failover/failovertests.py:636: DeprecationWarning: invalid escape sequence \ self.data_path = 'c:/Program\ Files/Couchbase/Server/var/lib/couchbase/data' test_failover_normal (failover.failovertests.FailoverTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_9 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t failover.failovertests.FailoverTests.test_failover_normal,replicas=1,graceful=False,num_failed_nodes=1,items=10000,active_resident_threshold=70,dgm_run=True,failoverMaster=True,GROUP=P0 Test Input params: {'replicas': '1', 'graceful': 'False', 'num_failed_nodes': '1', 'items': '10000', 'active_resident_threshold': '70', 'dgm_run': 'True', 'failoverMaster': 'True', 'GROUP': 'P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 9, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_9'} [2022-03-01 04:09:38,523] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:38,658] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:38,932] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:39,316] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:39,325] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:09:39,345] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 04:09:39,345] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #9 test_failover_normal============== [2022-03-01 04:09:39,345] - [basetestcase:331] INFO - initializing cluster [2022-03-01 04:09:39,529] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:09:39,535] - [task:159] INFO - {'uptime': '3534', 'memoryTotal': 3947372544, 'memoryFree': 2661801984, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:09:39,543] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 04:09:39,543] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 04:09:39,543] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 04:09:39,553] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 04:09:39,553] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 04:09:39,553] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 04:09:39,564] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 04:09:39,575] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 04:09:39,575] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 04:09:39,577] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 04:09:39,577] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 04:09:39,577] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:09:39,578] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 04:09:39,701] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:09:39,702] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:39,839] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:09:40,107] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:40,488] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:40,489] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:09:40,529] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:40,529] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:09:40,533] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:40,535] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:40,542] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:09:40,587] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:09:40,592] - [task:159] INFO - {'uptime': '3532', 'memoryTotal': 3947372544, 'memoryFree': 3236372480, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:09:40,594] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:09:40,595] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 04:09:40,601] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:09:40,601] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 04:09:40,755] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:09:40,758] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:40,896] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:09:41,168] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:41,550] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:41,551] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:09:41,629] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:41,629] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:09:41,632] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:41,636] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:41,642] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:09:41,665] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:09:41,668] - [task:159] INFO - {'uptime': '3508', 'memoryTotal': 4201627648, 'memoryFree': 3561889792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:09:41,670] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:09:41,670] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:09:41,673] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:09:41,674] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 04:09:41,740] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:09:41,742] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:41,851] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:09:42,164] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:42,494] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:42,494] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:09:42,564] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:42,564] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:09:42,567] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:42,569] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:42,573] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:09:42,594] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:09:42,597] - [task:159] INFO - {'uptime': '3511', 'memoryTotal': 4201627648, 'memoryFree': 3575173120, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:09:42,599] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:09:42,599] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:09:42,602] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:09:42,603] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 04:09:42,668] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:09:42,670] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:42,777] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:09:43,141] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:43,485] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:43,486] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:09:43,555] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:43,556] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:09:43,559] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:43,561] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:43,565] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:09:43,579] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:09:43,582] - [task:159] INFO - {'uptime': '3511', 'memoryTotal': 4201627648, 'memoryFree': 3522961408, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:09:43,584] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:09:43,585] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:09:43,588] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:09:43,588] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 04:09:43,655] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:09:43,657] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:43,764] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:09:44,058] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:44,393] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:44,394] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:09:44,463] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:44,463] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:09:44,466] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:44,468] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:44,472] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:09:44,484] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:09:44,487] - [task:159] INFO - {'uptime': '3511', 'memoryTotal': 4201627648, 'memoryFree': 3522912256, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:09:44,489] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:09:44,489] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:09:44,492] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:09:44,492] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 04:09:44,555] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:09:44,557] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:09:44,665] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:09:44,975] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:09:45,308] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:09:45,309] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:09:45,379] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:09:45,379] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:09:45,383] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:45,385] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:09:45,389] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:09:45,399] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 04:09:45,569] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 04:09:50,575] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 04:09:50,594] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 04:09:51,406] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 04:09:51,418] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:10:01,428] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-03-01 04:10:01,429] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:10:17,970] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 04:10:17,983] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:10:27,999] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:10:27,999] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:10:41,916] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 04:10:41,934] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:10:51,952] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:10:51,952] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:11:05,892] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 04:11:05,912] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:11:15,930] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:11:15,930] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:11:29,818] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 04:11:29,840] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:11:39,859] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:11:39,859] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:11:53,593] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 04:11:53,594] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 04:11:53,594] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 04:11:53,594] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 04:11:53,594] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 04:11:53,615] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:11:53,673] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:12:03,688] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:12:03,704] - [task:902] INFO - rebalancing was completed with progress: 100% in 10.030026912689209 sec [2022-03-01 04:12:03,706] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:03,848] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:12:04,125] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:04,508] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:04,509] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:12:04,586] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:12:04,586] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:12:04,587] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 04:12:04,740] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=256&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 04:12:04,775] - [rest_client:3014] INFO - 0.04 seconds to create bucket default [2022-03-01 04:12:04,775] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 04:12:05,583] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:12:05,736] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:12:05,908] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:12:06,035] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:12:06,220] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:12:06,364] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:12:06,713] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:12:06,952] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:12:07,081] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:12:07,219] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:12:07,382] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:12:07,494] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:12:08,316] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 256 [2022-03-01 04:12:08,318] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #9 test_failover_normal ============== [2022-03-01 04:12:08,334] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:08,471] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:12:08,753] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:09,132] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:09,135] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:09,275] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:12:09,574] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:09,963] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:09,966] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:10,074] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:12:10,398] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:10,724] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:10,726] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:10,833] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:12:11,140] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:11,479] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:11,481] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:11,588] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:12:11,878] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:12,220] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:12,223] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:12:12,327] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:12:12,651] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:12:12,987] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:12:25,183] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:12:25,183] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 7.635342185903983, 'mem_free': 2756395008, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:12:25,183] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.823947234906139, 'mem_free': 3348017152, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 04:12:25,183] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.800304491245877, 'mem_free': 3303108608, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 04:12:25,183] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 4.981036662452592, 'mem_free': 3328516096, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 04:12:25,183] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.858013716027432, 'mem_free': 3303272448, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 04:12:25,183] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.63289843949859, 'mem_free': 3026976768, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:12:25,184] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:12:25,184] - [basetestcase:559] INFO - Time to execute basesetup : 166.66311883926392 [2022-03-01 04:12:25,184] - [failoverbasetests:85] INFO - ============== FailoverBaseTest setup was finished for test #9 test_failover_normal ============== [2022-03-01 04:12:25,184] - [failovertests:49] INFO - Picking node 172.23.106.52 as reference node for test case [2022-03-01 04:12:25,184] - [failovertests:319] INFO - num_replicas : 1 [2022-03-01 04:12:25,184] - [failovertests:320] INFO - recoveryType : ['delta'] [2022-03-01 04:12:25,184] - [failovertests:321] INFO - failover_reason : normal [2022-03-01 04:12:25,185] - [failovertests:322] INFO - num_failed_nodes : 1 [2022-03-01 04:12:25,185] - [failovertests:323] INFO - picking server : ip:172.23.106.52 port:8091 ssh_username:nonroot as the master [2022-03-01 04:12:25,261] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:12:25,261] - [basetestcase:829] INFO - sleep for 120 secs. ... [2022-03-01 04:14:25,410] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:14:25,587] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:14:25,671] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:14:25,757] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:14:25,896] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:14:25,992] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:14:26,515] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:14:26,522] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:14:26,524] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:14:26,527] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:14:26,663] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:14:26,664] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:14:26,671] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:14:26,671] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:14:26,752] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:14:26,753] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:14:26,754] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:14:26,763] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:14:26,906] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:14:26,907] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:14:26,909] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:14:26,911] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:14:26,997] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:14:27,000] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:14:27,000] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:14:27,003] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:14:27,097] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:14:27,097] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:14:27,099] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:14:27,100] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:14:28,334] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:14:28,347] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:14:29,309] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:29,447] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:14:29,722] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:30,103] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:30,104] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:30,361] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:30,362] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:14:30,364] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:30,512] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:14:30,790] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:31,169] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:31,170] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:14:32,650] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:32,656] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:14:32,658] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:32,794] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:14:33,068] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:33,453] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:33,454] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:33,751] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:33,752] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:14:33,754] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:33,886] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:14:34,156] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:34,541] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:34,542] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:14:35,716] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:35,722] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:14:35,725] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:35,828] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:14:36,147] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:36,472] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:36,473] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:36,640] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:36,642] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:14:36,644] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:36,743] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:14:37,049] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:37,380] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:37,381] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:14:38,021] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:38,027] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:14:38,029] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:38,138] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:14:38,445] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:38,781] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:38,782] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:38,957] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:38,959] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:14:38,960] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:39,061] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:14:39,359] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:39,700] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:39,701] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:14:40,357] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:40,364] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:14:40,366] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:40,471] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:14:40,763] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:41,106] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:41,107] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:41,291] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:41,293] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:14:41,295] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:41,399] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:14:41,687] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:42,030] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:42,031] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:14:42,685] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:42,692] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:14:42,694] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:42,808] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:14:43,141] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:43,484] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:43,485] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:43,660] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:43,662] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:14:43,664] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:43,767] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:14:44,081] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:44,417] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:44,418] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:14:45,080] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:45,086] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:14:45,112] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:14:45,124] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:14:46,089] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:46,224] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:14:46,497] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:46,887] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:46,888] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:47,186] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:47,190] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:47,324] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:14:47,594] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:47,975] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:47,976] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:48,268] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:48,271] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:48,375] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:14:48,685] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:49,013] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:49,014] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:49,183] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:49,186] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:49,289] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:14:49,589] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:49,927] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:49,928] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:50,060] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:50,063] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:50,167] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:14:50,450] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:50,792] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:50,794] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:50,967] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:50,971] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:51,074] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:14:51,382] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:51,713] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:51,714] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:51,886] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:51,888] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:14:51,890] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:52,032] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:14:52,338] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:52,724] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:52,725] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:53,028] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:53,032] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:53,172] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:14:53,451] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:53,838] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:53,838] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:54,131] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:54,134] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:54,236] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:14:54,549] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:54,885] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:54,886] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:55,061] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:55,064] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:55,168] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:14:55,485] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:55,831] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:55,832] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:56,007] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:56,010] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:56,116] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:14:56,401] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:56,735] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:56,736] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:56,912] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:56,915] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:57,019] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:14:57,326] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:57,658] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:57,658] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:57,831] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:57,832] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:14:57,834] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:57,972] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:14:58,247] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:58,628] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:58,629] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:58,883] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:58,887] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:14:59,030] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:14:59,302] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:14:59,689] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:14:59,690] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:14:59,987] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:14:59,990] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:00,095] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:00,409] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:00,742] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:00,743] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:00,912] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:00,915] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:01,017] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:01,323] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:01,660] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:01,661] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:01,834] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:01,837] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:01,942] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:02,224] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:02,558] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:02,559] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:02,726] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:02,729] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:02,831] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:03,144] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:03,478] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:03,479] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:03,647] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:03,649] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:15:03,651] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:03,802] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:15:04,072] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:04,453] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:04,454] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:04,747] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:04,751] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:04,888] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:05,156] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:05,536] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:05,537] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:05,827] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:05,831] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:05,935] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:06,241] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:06,574] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:06,575] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:06,745] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:06,749] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:06,853] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:07,154] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:07,494] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:07,495] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:07,671] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:07,674] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:07,777] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:08,077] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:08,421] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:08,421] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:08,597] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:08,600] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:08,699] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:09,004] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:09,328] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:09,329] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:09,516] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:09,517] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:15:09,554] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:15:09,554] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:15:09,801] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:09,943] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:15:10,214] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:10,598] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:10,599] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:11,079] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:11,086] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:11,187] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:11,491] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:11,835] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:11,835] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:12,083] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:12,091] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:12,198] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:12,490] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:12,824] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:12,825] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:13,074] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:13,082] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:13,189] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:13,511] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:13,842] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:13,843] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:14,089] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:14,096] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:14,200] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:14,515] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:14,849] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:14,850] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:15,091] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:15,098] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:15,233] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:15,506] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:15,888] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:15,889] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:16,357] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:16,398] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:15:16,411] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:15:16,430] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:16,565] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:15:16,839] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:17,216] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:17,218] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:15:18,395] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:15:18,640] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:18,778] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:19,055] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:19,457] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:19,458] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:15:20,644] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:15:20,890] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:20,994] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:21,310] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:21,651] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:21,652] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:15:22,222] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:15:22,435] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:22,541] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:22,843] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:23,177] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:23,177] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:15:23,742] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:15:23,958] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:24,063] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:24,343] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:24,678] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:24,679] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:15:25,252] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:15:25,472] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:25,578] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:25,886] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:26,219] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:26,220] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:15:26,797] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:15:27,009] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:27,146] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:15:27,414] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:27,801] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:27,802] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:28,273] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:28,283] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:28,419] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:28,687] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:29,073] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:29,074] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:29,501] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:29,509] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:29,611] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:29,921] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:30,253] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:30,254] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:30,495] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:30,503] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:30,607] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:30,913] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:31,249] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:31,250] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:31,493] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:31,502] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:31,604] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:31,884] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:32,225] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:32,226] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:32,444] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:32,451] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:32,556] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:32,874] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:33,211] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:33,212] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:15:33,418] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:33,474] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:33,612] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:15:33,880] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:34,260] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:34,260] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:34,544] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:34,545] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:34,822] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:34,825] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:34,960] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:35,239] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:35,632] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:35,633] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:35,918] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:35,919] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:36,157] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:36,160] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:36,264] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:36,573] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:36,905] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:36,906] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:37,074] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:37,075] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:37,197] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:37,200] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:37,304] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:37,606] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:37,939] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:37,940] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:38,103] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:38,104] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:38,228] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:38,231] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:38,329] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:38,616] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:38,951] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:38,952] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:39,116] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:39,117] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:39,238] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:39,241] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:39,347] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:39,660] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:39,993] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:39,994] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:40,154] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:40,154] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:40,278] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:40,280] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:40,418] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:15:40,689] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:41,075] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:41,076] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:15:41,393] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:41,400] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:41,538] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:41,812] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:42,192] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:42,193] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:15:42,557] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:42,564] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:42,669] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:42,988] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:43,324] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:43,326] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:15:43,518] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:43,524] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:43,632] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:43,947] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:44,292] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:44,293] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:15:44,495] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:44,502] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:44,608] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:44,908] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:45,258] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:45,259] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:15:45,451] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:45,458] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:45,560] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:45,881] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:46,213] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:46,214] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:15:46,408] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:47,938] - [rest_client:1790] INFO - fail_over node ns_1@172.23.104.172 successful [2022-03-01 04:15:47,938] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 04:15:47,973] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:15:47,973] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:15:47,985] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:15:47,986] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:15:47,997] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:15:47,998] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:15:48,010] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:15:48,010] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:15:48,012] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:48,144] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:48,412] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:48,795] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:48,796] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:49,068] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:49,068] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:49,345] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:49,348] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:49,449] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:49,757] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:50,093] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:50,094] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:50,261] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:50,262] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:50,424] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:50,426] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:50,528] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:50,832] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:51,167] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:51,168] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:51,334] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:51,336] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:51,460] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:51,463] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:51,567] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:15:51,853] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:52,197] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:52,198] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:52,370] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:52,371] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:52,497] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:52,500] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:52,605] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:15:52,909] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:53,243] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:53,244] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:53,400] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:53,400] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:15:53,520] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:53,522] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 04:15:53,522] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 04:15:53,543] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:15:53,544] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:15:53,555] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:15:53,556] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:15:53,569] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:53,707] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:53,973] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:54,362] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:54,363] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:54,665] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:54,667] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:15:54,669] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:54,817] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:15:55,086] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:55,463] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:55,463] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:15:56,679] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:56,684] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:15:56,686] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:56,790] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:57,113] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:57,451] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:57,452] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:57,626] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:57,627] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:15:57,629] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:57,731] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:15:58,048] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:58,388] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:58,389] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:15:58,935] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:58,941] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:15:58,943] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:59,048] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:15:59,356] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:15:59,689] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:15:59,690] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:15:59,866] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:15:59,867] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:15:59,869] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:15:59,976] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:16:00,283] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:00,623] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:00,624] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:01,157] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:01,161] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:16:01,163] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:01,270] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:16:01,561] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:01,897] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:01,898] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:02,072] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:02,074] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:16:02,076] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:02,175] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:16:02,455] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:02,788] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:02,788] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:03,330] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:03,336] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:16:03,338] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:03,442] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:16:03,758] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:04,093] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:04,094] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:04,266] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:04,267] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:16:04,269] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:04,368] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:16:04,663] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:04,992] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:04,993] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:05,524] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:05,530] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:16:05,530] - [basetestcase:829] INFO - sleep for 5 secs. after failover before invoking rebalance... ... [2022-03-01 04:16:10,536] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.104.172', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:16:10,598] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:16:10,602] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 04:16:30,627] - [rest_client:1951] INFO - rebalance progress took 20.03 seconds [2022-03-01 04:16:30,627] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:16:40,637] - [failovertests:197] INFO - Begin VERIFICATION for Rebalance after Failover Only [2022-03-01 04:16:40,660] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:16:40,672] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:16:40,685] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:16:40,696] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:16:41,574] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:41,711] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:16:41,977] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:42,363] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:42,364] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:42,660] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:42,661] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:16:42,664] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:42,796] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:16:43,063] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:43,444] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:43,445] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:45,128] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:45,135] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:16:45,137] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:45,241] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:16:45,552] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:45,891] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:45,892] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:46,070] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:46,072] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:16:46,074] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:46,173] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:16:46,484] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:46,822] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:46,823] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:47,558] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:47,567] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:16:47,569] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:47,672] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:16:47,983] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:48,329] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:48,330] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:48,509] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:48,511] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:16:48,513] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:48,616] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:16:48,922] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:49,268] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:49,269] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:50,002] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:50,009] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:16:50,011] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:50,115] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:16:50,405] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:50,737] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:50,738] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:50,910] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:50,911] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:16:50,913] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:51,012] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:16:51,297] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:51,632] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:51,633] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:52,346] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:52,354] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:16:52,356] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:52,463] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:16:52,767] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:53,098] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:53,099] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:16:53,273] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:53,274] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:16:53,276] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:53,379] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:16:53,691] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:16:54,024] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:16:54,025] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:16:54,767] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:16:54,774] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:16:54,926] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:16:55,005] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:16:55,082] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:16:55,216] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:16:55,299] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:16:55,443] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:16:55,777] - [task:2328] INFO - 0 items were verified [2022-03-01 04:16:59,160] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:16:59,160] - [task:2332] INFO - 10000 items were verified in 3.7164154052734375 sec.the average number of ops - 2690.0 per second [2022-03-01 04:16:59,186] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:16:59,199] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:16:59,806] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:16:59,944] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:17:00,223] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:00,605] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:00,606] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:00,896] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:00,899] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:01,005] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:17:01,330] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:01,669] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:01,670] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:01,844] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:01,848] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:01,951] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:17:02,258] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:02,597] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:02,598] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:02,773] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:02,775] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:02,874] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:17:03,148] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:03,476] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:03,477] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:03,648] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:03,651] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:03,756] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:17:04,078] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:04,421] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:04,422] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:04,603] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:04,605] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:17:04,606] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:04,746] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:17:05,018] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:05,411] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:05,412] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:05,705] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:05,708] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:05,812] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:17:06,117] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:06,451] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:06,452] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:06,623] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:06,626] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:06,730] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:17:07,036] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:07,370] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:07,371] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:07,543] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:07,546] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:07,651] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:17:07,930] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:08,262] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:08,263] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:08,436] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:08,439] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:08,540] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:17:08,839] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:09,172] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:09,172] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:09,345] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:09,346] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:17:09,348] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:09,487] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:17:09,754] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:10,139] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:10,140] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:10,397] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:10,400] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:10,502] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:17:10,808] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:11,142] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:11,143] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:11,311] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:11,314] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:11,420] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:17:11,735] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:12,082] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:12,083] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:12,257] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:12,260] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:12,361] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:17:12,657] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:13,006] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:13,007] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:13,150] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:13,155] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:13,262] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:17:13,568] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:13,908] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:13,908] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:14,052] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:14,054] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:17:14,056] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:14,191] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:17:14,469] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:14,849] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:14,850] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:15,139] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:15,142] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:15,244] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:17:15,554] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:15,883] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:15,884] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:16,058] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:16,061] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:16,164] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:17:16,471] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:16,808] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:16,809] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:16,986] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:16,989] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:17,089] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:17:17,371] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:17,703] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:17,704] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:17,875] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:17,878] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:17:17,983] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:17:18,284] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:17:18,614] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:17:18,614] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:17:18,787] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:17:18,789] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:17:18,789] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:17:19,885] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1994 [2022-03-01 04:17:19,886] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 04:17:19,886] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 205 [2022-03-01 04:17:19,886] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 205 [2022-03-01 04:17:19,887] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 2013 [2022-03-01 04:17:19,887] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 04:17:19,887] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 04:17:19,887] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 04:17:19,887] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 1987 [2022-03-01 04:17:19,887] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 04:17:19,887] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 04:17:19,887] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 04:17:19,888] - [rebalance_helper:370] INFO - curr_items from 172.23.105.162:8091 : 1980 [2022-03-01 04:17:19,888] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.162:8091 : 0 [2022-03-01 04:17:19,888] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.162:8091 : 205 [2022-03-01 04:17:19,888] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.162:8091 : 205 [2022-03-01 04:17:19,888] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 2026 [2022-03-01 04:17:19,888] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 04:17:19,888] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 04:17:19,888] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 04:17:19,888] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 04:17:19,889] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:17:20,269] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 04:17:20,269] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:17:20,270] - [basetestcase:829] INFO - sleep for 60 secs. ... [2022-03-01 04:18:20,331] - [basetestcase:2135] INFO - Begin Verification for data comparison [2022-03-01 04:18:20,357] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:18:20,369] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:18:20,384] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:20,518] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:18:20,867] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:21,253] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:21,254] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:18:22,512] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:18:22,764] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:22,871] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:18:23,180] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:23,513] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:23,514] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:18:24,081] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:18:24,303] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:24,408] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:18:24,721] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:25,062] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:25,063] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:18:25,660] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:18:25,887] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:25,991] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:18:26,280] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:26,622] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:26,623] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:18:27,207] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:18:27,432] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:27,536] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:18:27,860] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:28,194] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:28,195] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:18:28,753] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:18:29,021] - [basetestcase:2144] INFO - End Verification for data comparison [2022-03-01 04:18:29,076] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 04:18:29,087] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:18:29,098] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:18:29,109] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:18:29,120] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:18:29,121] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:29,226] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:18:29,528] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:29,869] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:29,870] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:30,038] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:30,039] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:30,168] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:30,171] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:30,274] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:18:30,562] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:30,909] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:30,910] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:31,077] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:31,078] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:31,244] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:31,247] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:31,351] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:18:31,660] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:31,997] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:31,999] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:32,165] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:32,166] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:32,294] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:32,297] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:32,398] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:18:32,710] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:33,044] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:33,045] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:33,209] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:33,210] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:33,334] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:33,337] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:33,475] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:18:33,743] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:34,131] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:34,132] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:34,414] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:34,414] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:18:34,656] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:34,657] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 04:18:34,716] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 04:18:34,717] - [failovertests:219] INFO - End VERIFICATION for Rebalance after Failover Only [2022-03-01 04:18:34,719] - [failoverbasetests:98] INFO - ============== tearDown was started for test #9 test_failover_normal ============== [2022-03-01 04:18:34,721] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:34,863] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:18:35,142] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:35,519] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:35,520] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:18:35,637] - [remote_util:177] INFO - process beam.smp is running on 172.23.104.172: with pid 38991 [2022-03-01 04:18:35,637] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2022-03-01 04:18:35,770] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2022-03-01 04:18:35,770] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:18:35,770] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -F [2022-03-01 04:18:35,950] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:35,950] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -t nat -F [2022-03-01 04:18:36,010] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:36,010] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:36,165] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:18:36,166] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:18:36,166] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT memcached [2022-03-01 04:18:36,372] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:18:36,372] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:18:36,372] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT beam [2022-03-01 04:18:36,408] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:18:36,408] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:18:36,410] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:36,551] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:18:36,811] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:37,195] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:37,195] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:18:37,279] - [remote_util:177] INFO - process beam.smp is running on 172.23.106.52: with pid 27929 [2022-03-01 04:18:37,280] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 [2022-03-01 04:18:37,416] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as root [2022-03-01 04:18:37,416] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:18:37,416] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -F [2022-03-01 04:18:37,597] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:37,598] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -t nat -F [2022-03-01 04:18:37,662] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:37,662] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:37,793] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:18:37,794] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:18:37,794] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT memcached [2022-03-01 04:18:38,022] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:18:38,023] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:18:38,023] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT beam [2022-03-01 04:18:38,060] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:18:38,060] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:18:38,062] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:38,164] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:18:38,475] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:38,809] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:38,810] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:18:38,899] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.158: with pid 25310 [2022-03-01 04:18:38,899] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 [2022-03-01 04:18:39,001] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as root [2022-03-01 04:18:39,001] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:18:39,001] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -F [2022-03-01 04:18:39,229] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:39,229] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -t nat -F [2022-03-01 04:18:39,285] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:39,285] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:39,389] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:18:39,389] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:18:39,390] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT memcached [2022-03-01 04:18:39,638] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:18:39,638] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:18:39,639] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT beam [2022-03-01 04:18:39,701] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:18:39,701] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:18:39,703] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:39,811] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:18:40,127] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:40,471] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:40,472] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:18:40,558] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.126: with pid 10785 [2022-03-01 04:18:40,558] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 [2022-03-01 04:18:40,656] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as root [2022-03-01 04:18:40,656] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:18:40,656] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -F [2022-03-01 04:18:40,870] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:40,870] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -t nat -F [2022-03-01 04:18:40,926] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:40,927] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:41,028] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:18:41,029] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:18:41,029] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT memcached [2022-03-01 04:18:41,270] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:18:41,270] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:18:41,270] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT beam [2022-03-01 04:18:41,334] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:18:41,335] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:18:41,337] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:41,441] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:18:41,724] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:42,063] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:42,064] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:18:42,149] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.127: with pid 11088 [2022-03-01 04:18:42,149] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 [2022-03-01 04:18:42,249] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as root [2022-03-01 04:18:42,250] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:18:42,250] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -F [2022-03-01 04:18:42,446] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:42,446] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -t nat -F [2022-03-01 04:18:42,500] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:42,501] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:42,604] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:18:42,604] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:18:42,604] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT memcached [2022-03-01 04:18:42,840] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:18:42,840] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:18:42,841] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT beam [2022-03-01 04:18:42,905] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:18:42,905] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:18:42,907] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:43,010] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:18:43,323] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:18:43,659] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:18:43,660] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:18:43,746] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.162: with pid 43223 [2022-03-01 04:18:43,747] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 [2022-03-01 04:18:43,847] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as root [2022-03-01 04:18:43,847] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:18:43,847] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -F [2022-03-01 04:18:44,067] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:44,067] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -t nat -F [2022-03-01 04:18:44,122] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:18:44,122] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:18:44,224] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:18:44,224] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:18:44,224] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT memcached [2022-03-01 04:18:44,475] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:18:44,475] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:18:44,476] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT beam [2022-03-01 04:18:44,537] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:18:44,537] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:18:54,551] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 04:18:54,587] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.106.52 [2022-03-01 04:18:55,444] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 04:18:55,450] - [rest_client:140] INFO - node 172.23.106.52 existing buckets : [] [2022-03-01 04:18:55,450] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.106.52 [2022-03-01 04:18:55,458] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:18:55,466] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:18:55,473] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:18:55,482] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:18:55,489] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:18:55,496] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:18:55,511] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:18:55,546] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 04:18:55,548] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:18:55,608] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:18:55,612] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 04:19:05,627] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:19:15,643] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:19:35,671] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 04:19:35,671] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:19:45,706] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.105.162', 8091)] [2022-03-01 04:19:45,709] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:19:45,715] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:19:45,723] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:19:45,728] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:19:45,737] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:19:45,741] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:19:45,750] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:19:45,754] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:19:45,760] - [failoverbasetests:109] INFO - ============== tearDown was finished for test #9 test_failover_normal ============== [2022-03-01 04:19:45,777] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:19:45,778] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.085310449268047, 'mem_free': 2949582848, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:19:45,778] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:19:45,779] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:19:45,919] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:19:46,191] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:19:46,564] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:19:46,566] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:19:46,707] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:19:46,975] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:19:47,365] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:19:47,369] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:19:47,475] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:19:47,797] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:19:48,129] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:19:48,132] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:19:48,241] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:19:48,543] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:19:48,871] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:19:48,873] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:19:48,979] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:19:49,261] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:19:49,603] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:19:49,607] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:19:49,710] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:19:50,025] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:19:50,358] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:02,677] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #9 test_failover_normal ============== [2022-03-01 04:20:02,708] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 04:20:02,718] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 04:20:02,725] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:20:02,731] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:20:02,736] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:20:02,742] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:20:02,742] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 04:20:02,755] - [rest_client:1073] ERROR - DELETE http://172.23.106.52:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 04:20:02,755] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 04:20:02,761] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,784] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 04:20:02,784] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,788] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 04:20:02,795] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 04:20:02,795] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,800] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 04:20:02,804] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 04:20:02,804] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,807] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 04:20:02,810] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 04:20:02,810] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,813] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 04:20:02,816] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 04:20:02,816] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,818] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 04:20:02,821] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 04:20:02,821] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:02,824] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 04:20:02,824] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #9 test_failover_normal ============== [2022-03-01 04:20:02,824] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 04:20:02,825] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 summary so far suite failover.failovertests.FailoverTests , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_9 ok ---------------------------------------------------------------------- Ran 1 test in 624.311s OK test_failover_stop_server (failover.failovertests.FailoverTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_10 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t failover.failovertests.FailoverTests.test_failover_stop_server,replicas=1,graceful=False,num_failed_nodes=1,numViews=1,withViewsOps=True,createIndexesDuringFailover=True,items=10000,active_resident_threshold=70,dgm_run=True,failoverMaster=True,GROUP=P0 Test Input params: {'replicas': '1', 'graceful': 'False', 'num_failed_nodes': '1', 'numViews': '1', 'withViewsOps': 'True', 'createIndexesDuringFailover': 'True', 'items': '10000', 'active_resident_threshold': '70', 'dgm_run': 'True', 'failoverMaster': 'True', 'GROUP': 'P0', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 10, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_10'} [2022-03-01 04:20:02,847] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:03,005] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:20:03,281] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:03,663] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:03,670] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:20:03,681] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 04:20:03,681] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #10 test_failover_stop_server============== [2022-03-01 04:20:03,682] - [basetestcase:331] INFO - initializing cluster [2022-03-01 04:20:03,852] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:20:03,856] - [task:159] INFO - {'uptime': '4154', 'memoryTotal': 3947372544, 'memoryFree': 3187298304, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:20:03,859] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:20:03,859] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 04:20:03,859] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 04:20:03,860] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 04:20:03,864] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 04:20:03,865] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 04:20:03,865] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 04:20:03,870] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 04:20:03,874] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 04:20:03,875] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 04:20:03,903] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:20:03,903] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 04:20:04,101] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:20:04,103] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:04,246] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:20:04,531] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:04,991] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:04,992] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:20:05,044] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:20:05,045] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:20:05,052] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:05,055] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:05,060] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:20:05,088] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:20:05,095] - [task:159] INFO - {'uptime': '4155', 'memoryTotal': 3947372544, 'memoryFree': 2975379456, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:20:05,104] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 04:20:05,123] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:20:05,123] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 04:20:05,254] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:20:05,256] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:05,402] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:20:05,680] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:06,067] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:06,068] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:20:06,147] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:20:06,147] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:20:06,151] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:06,155] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:06,167] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:20:06,199] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:20:06,202] - [task:159] INFO - {'uptime': '4130', 'memoryTotal': 4201627648, 'memoryFree': 3559129088, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:20:06,203] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:20:06,204] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:20:06,207] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:20:06,207] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 04:20:06,266] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:20:06,268] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:06,372] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:20:06,693] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:07,033] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:07,034] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:20:07,102] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:20:07,102] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:20:07,105] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:07,107] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:07,110] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:20:07,122] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:20:07,125] - [task:159] INFO - {'uptime': '4132', 'memoryTotal': 4201627648, 'memoryFree': 3574657024, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:20:07,127] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:20:07,127] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:20:07,130] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:20:07,130] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 04:20:07,192] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:20:07,193] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:07,298] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:20:07,603] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:07,942] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:07,943] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:20:08,015] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:20:08,015] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:20:08,018] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:08,020] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:08,025] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:20:08,038] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:20:08,042] - [task:159] INFO - {'uptime': '4136', 'memoryTotal': 4201627648, 'memoryFree': 3530268672, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:20:08,044] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:20:08,044] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:20:08,048] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:20:08,048] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 04:20:08,106] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:20:08,108] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:08,208] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:20:08,489] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:08,821] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:08,822] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:20:08,889] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:20:08,889] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:20:08,892] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:08,894] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:08,898] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:20:08,912] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:20:08,915] - [task:159] INFO - {'uptime': '4136', 'memoryTotal': 4201627648, 'memoryFree': 3531423744, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:20:08,917] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:20:08,917] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:20:08,920] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:20:08,921] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 04:20:08,982] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:20:08,984] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:20:09,093] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:20:09,399] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:20:09,723] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:20:09,724] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:20:09,791] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:20:09,792] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:20:09,795] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:09,797] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:20:09,801] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:20:09,812] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 04:20:09,829] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 04:20:09,830] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2022-03-01 04:20:09,964] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 04:20:14,969] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 04:20:14,984] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 04:20:15,819] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 04:20:15,828] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:20:25,844] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:20:25,845] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:20:43,242] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 04:20:43,255] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:20:53,272] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:20:53,272] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:21:08,273] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 04:21:08,290] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:21:18,299] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-03-01 04:21:18,299] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:21:32,069] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 04:21:32,089] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:21:42,106] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:21:42,107] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:21:55,709] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 04:21:55,730] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:22:05,747] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:22:05,747] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:22:19,651] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 04:22:19,651] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 04:22:19,651] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 04:22:19,651] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 04:22:19,651] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 04:22:19,671] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:22:19,728] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:22:29,743] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:22:29,755] - [task:902] INFO - rebalancing was completed with progress: 100% in 10.026102542877197 sec [2022-03-01 04:22:29,757] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:29,892] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:22:30,164] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:30,550] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:30,551] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:22:30,632] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:22:30,632] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:22:30,632] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 04:22:30,786] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=256&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 04:22:30,819] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 04:22:30,819] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 04:22:31,589] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:22:31,787] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:22:31,911] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:22:32,102] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:22:32,241] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:22:32,386] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:22:32,789] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:22:32,940] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:22:33,096] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:22:33,294] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:22:33,438] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:22:33,566] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:22:34,434] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 256 [2022-03-01 04:22:34,495] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #10 test_failover_stop_server ============== [2022-03-01 04:22:34,507] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:34,639] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:22:34,905] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:35,278] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:35,281] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:35,418] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:22:35,695] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:36,096] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:36,099] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:36,207] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:22:36,532] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:36,860] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:36,864] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:36,967] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:22:37,273] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:37,605] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:37,607] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:37,713] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:22:38,002] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:38,340] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:38,342] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:22:38,450] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:22:38,763] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:22:39,090] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:22:51,262] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:22:51,262] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 24.32293010059324, 'mem_free': 3010678784, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:22:51,263] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.275113579000505, 'mem_free': 3408162816, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 04:22:51,263] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.943462897526502, 'mem_free': 3377725440, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 04:22:51,263] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.904713634059807, 'mem_free': 3404300288, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 04:22:51,263] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.091733870967742, 'mem_free': 3377639424, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 04:22:51,263] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 8.61850443599493, 'mem_free': 3086102528, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:22:51,263] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:22:51,263] - [basetestcase:559] INFO - Time to execute basesetup : 168.417986869812 [2022-03-01 04:22:51,264] - [failoverbasetests:85] INFO - ============== FailoverBaseTest setup was finished for test #10 test_failover_stop_server ============== [2022-03-01 04:22:51,264] - [failovertests:49] INFO - Picking node 172.23.106.52 as reference node for test case [2022-03-01 04:22:51,264] - [failovertests:319] INFO - num_replicas : 1 [2022-03-01 04:22:51,264] - [failovertests:320] INFO - recoveryType : ['delta'] [2022-03-01 04:22:51,264] - [failovertests:321] INFO - failover_reason : stop_server [2022-03-01 04:22:51,264] - [failovertests:322] INFO - num_failed_nodes : 1 [2022-03-01 04:22:51,264] - [failovertests:323] INFO - picking server : ip:172.23.106.52 port:8091 ssh_username:nonroot as the master [2022-03-01 04:22:51,329] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:22:51,329] - [basetestcase:829] INFO - sleep for 120 secs. ... [2022-03-01 04:24:51,521] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:24:51,638] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:24:51,711] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:24:51,842] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:24:51,918] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:24:51,995] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:24:52,510] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:24:52,511] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:24:52,511] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:24:52,514] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:24:52,645] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:24:52,646] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:24:52,646] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:24:52,647] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:24:52,818] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:24:52,818] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:24:52,820] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:24:52,821] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:24:52,889] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:24:52,889] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:24:52,889] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:24:52,889] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:24:52,964] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:24:52,965] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:24:52,966] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:24:52,971] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:24:53,123] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:24:53,127] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:24:53,128] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:24:53,129] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:24:54,140] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:24:54,151] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:24:54,381] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:24:54,544] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:24:54,833] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:24:55,217] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:24:55,218] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:24:55,515] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:24:55,517] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:24:55,519] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:24:55,652] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:24:55,919] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:24:56,295] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:24:56,296] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:24:57,759] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:24:57,765] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:24:57,767] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:24:57,906] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:24:58,182] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:24:58,566] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:24:58,567] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:24:58,863] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:24:58,865] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:24:58,867] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:24:59,006] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:24:59,272] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:24:59,653] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:24:59,654] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:25:01,117] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:01,123] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:25:01,125] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:01,227] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:01,540] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:01,874] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:01,874] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:02,049] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:02,050] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:25:02,052] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:02,154] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:02,465] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:02,802] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:02,803] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:25:03,455] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:03,462] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:25:03,464] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:03,570] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:03,887] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:04,239] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:04,240] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:04,420] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:04,422] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:25:04,425] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:04,528] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:04,838] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:05,217] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:05,218] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:25:05,878] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:05,884] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:25:05,887] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:05,993] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:06,288] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:06,624] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:06,625] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:06,799] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:06,800] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:25:06,803] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:06,901] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:07,184] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:07,516] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:07,516] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:25:08,161] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:08,167] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:25:08,169] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:08,273] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:08,581] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:08,915] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:08,916] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:09,089] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:09,091] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:25:09,093] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:09,192] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:09,494] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:09,825] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:09,826] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:25:10,477] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:10,483] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:25:10,507] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:25:10,519] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:25:11,486] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:11,620] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:11,888] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:12,269] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:12,270] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:12,563] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:12,566] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:12,712] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:12,986] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:13,368] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:13,369] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:13,624] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:13,628] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:13,732] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:14,045] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:14,383] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:14,384] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:14,519] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:14,522] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:14,627] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:14,938] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:15,286] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:15,286] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:15,465] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:15,469] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:15,575] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:15,867] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:16,213] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:16,214] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:16,386] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:16,389] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:16,494] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:16,807] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:17,148] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:17,148] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:17,323] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:17,325] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:25:17,327] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:17,461] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:17,737] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:18,115] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:18,116] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:18,417] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:18,420] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:18,561] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:18,841] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:19,226] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:19,227] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:19,524] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:19,528] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:19,632] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:19,943] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:20,279] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:20,280] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:20,413] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:20,417] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:20,516] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:20,816] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:21,158] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:21,159] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:21,332] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:21,335] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:21,438] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:21,722] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:22,058] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:22,059] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:22,233] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:22,236] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:22,339] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:22,643] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:22,977] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:22,978] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:23,152] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:23,153] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:25:23,155] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:23,292] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:23,570] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:23,944] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:23,945] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:24,200] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:24,203] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:24,337] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:24,607] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:24,987] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:24,988] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:25,280] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:25,284] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:25,385] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:25,691] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:26,028] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:26,028] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:26,202] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:26,205] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:26,311] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:26,607] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:26,941] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:26,941] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:27,120] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:27,123] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:27,228] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:27,511] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:27,849] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:27,850] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:28,024] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:28,027] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:28,129] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:28,458] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:28,796] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:28,797] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:28,978] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:28,979] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:25:28,981] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:29,115] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:29,391] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:29,774] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:29,775] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:30,067] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:30,071] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:30,211] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:30,495] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:30,874] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:30,875] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:31,172] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:31,176] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:31,275] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:31,587] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:31,924] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:31,925] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:32,100] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:32,103] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:32,206] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:32,599] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:32,937] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:32,938] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:33,110] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:33,114] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:33,216] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:33,498] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:33,837] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:33,838] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:34,008] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:34,011] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:34,116] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:34,428] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:34,764] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:34,765] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:25:34,941] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:34,942] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:25:35,979] - [rest_client:1073] ERROR - GET http://172.23.106.52:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2022-03-01 04:25:35,979] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:25:36,045] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:25:36,223] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:25:36,392] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:25:36,589] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:25:36,818] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view0?stale=ok [2022-03-01 04:25:37,061] - [task:2865] INFO - view : default_view0 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:25:37,065] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view1?stale=ok [2022-03-01 04:25:37,088] - [task:2865] INFO - view : default_view1 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:25:37,091] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view2?stale=ok [2022-03-01 04:25:37,113] - [task:2865] INFO - view : default_view2 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:25:37,117] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view3?stale=ok [2022-03-01 04:25:37,140] - [task:2865] INFO - view : default_view3 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:25:37,143] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view4?stale=ok [2022-03-01 04:25:37,166] - [task:2865] INFO - view : default_view4 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:25:37,191] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:25:37,191] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:25:37,351] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:37,486] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:37,765] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:38,146] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:38,147] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:38,636] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:38,644] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:38,745] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:39,065] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:39,412] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:39,412] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:39,666] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:39,674] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:39,781] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:40,075] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:40,418] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:40,419] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:40,676] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:40,683] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:40,788] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:41,113] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:41,451] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:41,452] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:41,708] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:41,716] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:41,826] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:42,152] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:42,489] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:42,490] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:42,739] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:42,747] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:42,882] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:43,155] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:43,543] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:43,544] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:44,011] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:44,044] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:25:44,055] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:25:44,071] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:44,211] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:44,482] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:44,863] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:44,864] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:25:46,034] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:25:46,285] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:46,418] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:46,689] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:47,070] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:47,071] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:25:48,256] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:25:48,504] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:48,605] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:48,911] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:49,245] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:49,246] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:25:49,821] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:25:50,031] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:50,132] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:50,426] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:50,760] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:50,761] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:25:51,330] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:25:51,534] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:51,637] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:51,925] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:52,268] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:52,269] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:25:52,833] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:25:53,051] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:53,157] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:25:53,465] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:53,801] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:53,802] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:25:54,373] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:25:54,587] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:54,729] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:25:55,003] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:55,383] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:55,384] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:55,855] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:55,863] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:56,003] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:25:56,274] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:56,661] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:56,662] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:57,140] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:57,148] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:57,248] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:25:57,551] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:57,881] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:57,883] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:58,127] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:58,135] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:58,239] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:25:58,541] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:58,875] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:58,875] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:25:59,121] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:25:59,129] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:25:59,233] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:25:59,519] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:25:59,856] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:25:59,857] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:26:00,106] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:00,114] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:00,215] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:26:00,527] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:00,859] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:00,859] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:26:01,103] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:01,161] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:01,301] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:26:01,580] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:01,964] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:01,965] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:02,254] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:02,255] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:02,493] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:02,496] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:02,637] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:26:02,909] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:03,292] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:03,293] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:03,572] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:03,573] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:03,813] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:03,816] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:03,921] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:26:04,229] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:04,566] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:04,567] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:04,733] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:04,734] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:04,903] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:04,906] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:05,013] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:26:05,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:05,668] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:05,669] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:05,839] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:05,840] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:05,969] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:05,972] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:06,079] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:26:06,367] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:06,717] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:06,718] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:06,887] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:06,888] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:07,019] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:07,022] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:07,124] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:26:07,433] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:07,761] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:07,761] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:07,924] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:07,925] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:26:08,044] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:08,047] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:08,186] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:26:08,451] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:08,828] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:08,829] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:26:09,176] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:09,183] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:09,317] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:26:09,588] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:09,967] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:09,967] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:26:10,311] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:10,318] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:10,422] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:26:10,730] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:11,064] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:11,065] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:26:11,257] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:11,264] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:11,365] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:26:11,664] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:12,007] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:12,008] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:26:12,202] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:12,209] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:12,308] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:26:12,595] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:12,933] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:12,934] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:26:13,124] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:13,131] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:13,238] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:26:13,545] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:13,877] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:13,878] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:26:14,073] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:14,133] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:14,274] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:26:14,549] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:26:14,933] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:26:14,998] - [remote_util:981] INFO - non root couchbase installed at 172.23.104.172 [2022-03-01 04:26:14,999] - [remote_util:3902] INFO - Stop Couchbase Server with non root method [2022-03-01 04:26:14,999] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /home/nonroot//opt/couchbase/bin/couchbase-server -k [2022-03-01 04:26:16,921] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:26:16,921] - [basetestcase:2425] INFO - Couchbase stopped [2022-03-01 04:26:16,921] - [failovertests:425] INFO - 10 seconds delay to wait for membase-server to shutdown [2022-03-01 04:26:16,967] - [rest_client:157] INFO - node ns_1@172.23.104.172 status : unhealthy [2022-03-01 04:26:16,968] - [rest_client:164] INFO - node ns_1@172.23.104.172 status_reached : True [2022-03-01 04:26:17,258] - [task:4078] INFO - Failing over 172.23.104.172:8091 with graceful=False [2022-03-01 04:26:29,162] - [rest_client:1790] INFO - fail_over node ns_1@172.23.104.172 successful [2022-03-01 04:26:29,163] - [task:4058] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2022-03-01 04:26:29,172] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:31,066] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:31,778] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:32,552] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:33,310] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:34,044] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:34,718] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:26:34,718] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:26:34,731] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:35,454] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:26:35,455] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:26:35,468] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:36,298] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:26:36,299] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:26:36,312] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:37,106] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:26:37,106] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:26:37,119] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:26:37,777] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:26:37,777] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:26:38,789] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:26:38,792] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:26:38,795] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:26:38,799] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:26:38,802] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:26:48,818] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:26:48,819] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:26:58,850] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:26:58,851] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:26:58,862] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 04:26:58,862] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:26:59,828] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:26:59,968] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:27:00,247] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:00,629] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:00,630] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:00,922] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:00,923] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:27:00,926] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:01,063] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:27:01,334] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:01,719] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:01,720] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:02,917] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:02,923] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:27:02,925] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:03,035] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:27:03,343] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:03,676] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:03,677] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:03,853] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:03,854] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:27:03,856] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:03,958] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:27:04,266] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:04,609] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:04,610] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:05,160] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:05,166] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:27:05,168] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:05,271] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:27:05,572] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:05,912] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:05,913] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:06,085] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:06,087] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:27:06,090] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:06,193] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:27:06,503] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:06,843] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:06,843] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:07,393] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:07,399] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:27:07,401] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:07,504] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:27:07,790] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:08,137] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:08,138] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:08,312] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:08,314] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:27:08,315] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:08,415] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:27:08,697] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:09,033] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:09,034] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:09,571] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:09,576] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:27:09,579] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:09,686] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:27:09,993] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:10,331] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:10,332] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:10,502] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:10,503] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:27:10,505] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:10,610] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:27:10,910] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:11,243] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:11,244] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:11,791] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:11,796] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:27:11,797] - [basetestcase:829] INFO - sleep for 5 secs. after failover before invoking rebalance... ... [2022-03-01 04:27:16,802] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.104.172', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:27:16,856] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:27:17,812] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:21,717] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:22,534] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:23,306] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:24,058] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:24,933] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:25,645] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:27:25,646] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:27:25,658] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:26,413] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view1, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:27:26,413] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:27:26,425] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:27,194] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view2, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:27:27,194] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:27:27,207] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:27,962] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view3, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:27:27,963] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:27:27,975] - [rest_client:610] INFO - index query url: http://172.23.106.52:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:27:28,697] - [task:3077] INFO - Server: 172.23.106.52, Design Doc: dev_ddoc1, View: default_view4, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:27:28,697] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:27:29,708] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:27:29,711] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:27:29,714] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:27:29,717] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:27:29,721] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:27:39,738] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:27:39,738] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:27:49,748] - [failovertests:197] INFO - Begin VERIFICATION for Rebalance after Failover Only [2022-03-01 04:27:49,769] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:27:49,780] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:27:49,789] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:27:49,799] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:27:50,746] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:50,883] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:27:51,153] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:51,537] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:51,537] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:51,788] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:51,789] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:27:51,791] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:51,931] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:27:52,199] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:52,584] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:52,584] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:54,265] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:54,273] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:27:54,275] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:54,377] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:27:54,686] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:55,020] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:55,021] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:55,195] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:55,196] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:27:55,198] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:55,302] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:27:55,613] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:55,943] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:55,944] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:56,675] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:56,682] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:27:56,684] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:56,789] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:27:57,108] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:57,451] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:57,452] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:27:57,625] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:57,627] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:27:57,629] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:57,730] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:27:58,045] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:58,389] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:58,389] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:27:59,121] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:27:59,129] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:27:59,132] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:27:59,242] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:27:59,536] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:27:59,881] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:27:59,882] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:00,062] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:00,063] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:28:00,065] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:00,166] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:28:00,455] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:00,796] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:00,796] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:28:01,539] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:01,546] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:28:01,548] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:01,657] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:28:01,978] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:02,315] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:02,316] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:02,484] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:02,485] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:28:02,487] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:02,586] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:28:02,888] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:03,223] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:03,223] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:28:03,959] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:03,966] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:28:04,045] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:28:04,188] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:28:04,261] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:28:04,335] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:28:04,487] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:28:04,636] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:28:04,968] - [task:2328] INFO - 0 items were verified [2022-03-01 04:28:08,509] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:28:08,509] - [task:2332] INFO - 10000 items were verified in 3.8723537921905518 sec.the average number of ops - 2582.0 per second [2022-03-01 04:28:08,530] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:28:08,541] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:28:09,014] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:09,155] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:28:09,430] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:09,814] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:09,815] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:10,105] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:10,108] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:10,211] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:28:10,532] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:10,873] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:10,874] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:11,051] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:11,054] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:11,158] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:28:11,467] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:11,803] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:11,804] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:11,978] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:11,981] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:12,084] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:28:12,369] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:12,702] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:12,703] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:12,883] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:12,885] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:12,987] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:28:13,295] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:13,622] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:13,622] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:13,793] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:13,795] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:28:13,797] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:13,934] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:28:14,207] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:14,595] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:14,596] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:14,893] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:14,896] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:14,997] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:28:15,310] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:15,648] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:15,648] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:15,822] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:15,825] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:15,926] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:28:16,227] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:16,563] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:16,564] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:16,735] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:16,739] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:16,841] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:28:17,125] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:17,456] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:17,457] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:17,628] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:17,631] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:17,733] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:28:18,039] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:18,374] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:18,375] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:18,542] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:18,544] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:28:18,546] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:18,680] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:28:18,947] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:19,328] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:19,329] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:19,626] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:19,629] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:19,731] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:28:20,032] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:20,366] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:20,367] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:20,551] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:20,554] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:20,657] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:28:20,965] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:21,310] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:21,311] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:21,485] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:21,488] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:21,591] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:28:21,877] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:22,216] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:22,217] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:22,393] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:22,396] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:22,501] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:28:22,821] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:23,157] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:23,158] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:23,295] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:23,296] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:28:23,298] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:23,433] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:28:23,708] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:24,098] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:24,098] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:24,399] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:24,402] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:24,507] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:28:24,818] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:25,151] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:25,152] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:25,328] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:25,330] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:25,435] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:28:25,735] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:26,069] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:26,070] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:26,246] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:26,249] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:26,351] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:28:26,636] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:26,965] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:26,965] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:27,131] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:27,134] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:28:27,238] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:28:27,544] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:28:27,878] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:28:27,878] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:28:28,055] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:28:28,057] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:28:28,057] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:28:29,209] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1999 [2022-03-01 04:28:29,209] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 04:28:29,210] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 205 [2022-03-01 04:28:29,210] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 205 [2022-03-01 04:28:29,210] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 1999 [2022-03-01 04:28:29,210] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 04:28:29,210] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 04:28:29,210] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 04:28:29,210] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 1994 [2022-03-01 04:28:29,210] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 04:28:29,210] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 04:28:29,211] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 04:28:29,211] - [rebalance_helper:370] INFO - curr_items from 172.23.105.162:8091 : 2010 [2022-03-01 04:28:29,211] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.162:8091 : 0 [2022-03-01 04:28:29,211] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.162:8091 : 205 [2022-03-01 04:28:29,211] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.162:8091 : 205 [2022-03-01 04:28:29,211] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 1998 [2022-03-01 04:28:29,211] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 04:28:29,211] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 04:28:29,211] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 04:28:29,211] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 04:28:29,212] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:28:29,631] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 04:28:29,631] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:28:29,633] - [basetestcase:829] INFO - sleep for 60 secs. ... [2022-03-01 04:29:29,660] - [basetestcase:2135] INFO - Begin Verification for data comparison [2022-03-01 04:29:29,683] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:29:29,693] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:29:29,707] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:29,843] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:29:30,110] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:30,491] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:30,492] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:29:31,808] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:29:32,062] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:32,167] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:29:32,471] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:32,796] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:32,797] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:29:33,372] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:29:33,589] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:33,691] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:29:34,004] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:34,335] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:34,336] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:29:34,920] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:29:35,136] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:35,240] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:29:35,522] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:35,855] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:35,856] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:29:36,443] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:29:36,663] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:36,773] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:29:37,087] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:37,420] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:37,421] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:29:37,984] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:29:38,256] - [basetestcase:2144] INFO - End Verification for data comparison [2022-03-01 04:29:38,306] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 04:29:38,317] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:29:38,327] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:29:38,336] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:29:38,347] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:29:38,349] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:38,450] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:29:38,753] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:39,091] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:39,092] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:39,258] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:39,259] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:39,386] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:39,389] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:39,492] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:29:39,775] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:40,110] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:40,111] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:40,281] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:40,282] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:40,445] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:40,448] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:40,550] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:29:40,851] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:41,184] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:41,185] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:41,352] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:41,353] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:41,524] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:41,527] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:41,629] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:29:41,939] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:42,271] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:42,272] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:42,435] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:42,437] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:42,560] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:42,563] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:42,706] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:29:42,997] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:43,381] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:43,382] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:43,623] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:43,624] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:29:43,859] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:43,860] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 04:29:43,908] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 04:29:43,909] - [failovertests:219] INFO - End VERIFICATION for Rebalance after Failover Only [2022-03-01 04:29:43,912] - [failoverbasetests:98] INFO - ============== tearDown was started for test #10 test_failover_stop_server ============== [2022-03-01 04:29:43,914] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:44,050] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:29:44,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:44,709] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:44,710] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:44,825] - [remote_util:3931] INFO - Starting couchbase server [2022-03-01 04:29:44,825] - [remote_util:3941] INFO - Start Couchbase Server with non root method [2022-03-01 04:29:44,826] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /home/nonroot//opt/couchbase/bin/couchbase-server \-- -noinput -detached [2022-03-01 04:29:46,244] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:46,244] - [remote_util:3169] INFO - [2022-03-01 04:29:46,244] - [remote_util:3169] INFO - The maximum number of processes for the couchbase user is set too low (4096). [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - It must be at least 10000. [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - Normally this can be increased by adding the following lines to [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - /etc/security/limits.conf: [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - couchbase soft nproc [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - couchbase hard nproc [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - Where is greater than 10000. The procedure may be totally [2022-03-01 04:29:46,245] - [remote_util:3169] INFO - different if you're running so called "non-root/non-sudo install" or [2022-03-01 04:29:46,246] - [remote_util:3169] INFO - if you've built Couchbase Server from source. [2022-03-01 04:29:46,246] - [remote_util:3169] INFO - [2022-03-01 04:29:46,246] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:46,343] - [remote_util:177] INFO - process beam.smp is running on 172.23.104.172: with pid 130199 [2022-03-01 04:29:46,343] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2022-03-01 04:29:46,480] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2022-03-01 04:29:46,480] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:29:46,481] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -F [2022-03-01 04:29:46,665] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:46,666] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -t nat -F [2022-03-01 04:29:46,727] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:46,727] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:46,866] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:29:46,867] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:29:46,867] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT memcached [2022-03-01 04:29:47,075] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:29:47,075] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:29:47,076] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT beam [2022-03-01 04:29:47,113] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:29:47,113] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:29:47,115] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:47,249] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:29:47,516] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:47,907] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:47,908] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:47,998] - [remote_util:177] INFO - process beam.smp is running on 172.23.106.52: with pid 27929 [2022-03-01 04:29:47,998] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 [2022-03-01 04:29:48,143] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as root [2022-03-01 04:29:48,144] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:29:48,144] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -F [2022-03-01 04:29:48,328] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:48,329] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -t nat -F [2022-03-01 04:29:48,392] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:48,392] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:48,524] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:29:48,525] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:29:48,525] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT memcached [2022-03-01 04:29:48,733] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:29:48,734] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:29:48,734] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT beam [2022-03-01 04:29:48,774] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:29:48,774] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:29:48,776] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:48,880] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:29:49,189] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:49,525] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:49,525] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:49,615] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.158: with pid 25310 [2022-03-01 04:29:49,615] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 [2022-03-01 04:29:49,719] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as root [2022-03-01 04:29:49,720] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:29:49,720] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -F [2022-03-01 04:29:49,960] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:49,960] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -t nat -F [2022-03-01 04:29:50,018] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:50,018] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:50,123] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:29:50,123] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:29:50,123] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT memcached [2022-03-01 04:29:50,368] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:29:50,368] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:29:50,369] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT beam [2022-03-01 04:29:50,433] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:29:50,434] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:29:50,436] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:50,541] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:29:50,845] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:51,189] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:51,190] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:51,279] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.126: with pid 10785 [2022-03-01 04:29:51,279] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 [2022-03-01 04:29:51,382] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as root [2022-03-01 04:29:51,383] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:29:51,383] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -F [2022-03-01 04:29:51,601] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:51,602] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -t nat -F [2022-03-01 04:29:51,659] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:51,659] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:51,767] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:29:51,768] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:29:51,768] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT memcached [2022-03-01 04:29:52,006] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:29:52,006] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:29:52,007] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT beam [2022-03-01 04:29:52,072] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:29:52,072] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:29:52,074] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:52,177] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:29:52,459] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:52,791] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:52,791] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:52,837] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.127: with pid 21386 [2022-03-01 04:29:52,837] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 [2022-03-01 04:29:52,936] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as root [2022-03-01 04:29:52,936] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:29:52,936] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -F [2022-03-01 04:29:53,130] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:53,130] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -t nat -F [2022-03-01 04:29:53,185] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:53,185] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:53,287] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:29:53,287] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:29:53,287] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT memcached [2022-03-01 04:29:53,505] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:29:53,506] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:29:53,506] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT beam [2022-03-01 04:29:53,570] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:29:53,570] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:29:53,572] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:53,675] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:29:53,982] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:29:54,318] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:29:54,319] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:29:54,408] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.162: with pid 43223 [2022-03-01 04:29:54,408] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 [2022-03-01 04:29:54,510] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as root [2022-03-01 04:29:54,510] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:29:54,510] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -F [2022-03-01 04:29:54,736] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:54,737] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -t nat -F [2022-03-01 04:29:54,793] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:29:54,793] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:29:54,896] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:29:54,896] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:29:54,896] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT memcached [2022-03-01 04:29:55,133] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:29:55,133] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:29:55,133] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT beam [2022-03-01 04:29:55,198] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:29:55,198] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:30:05,239] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 04:30:12,835] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/pools/default/buckets/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'["Unexpected server error, request logged."]' auth: Administrator:password [2022-03-01 04:30:12,836] - [rest_client:2875] WARNING - Bucket deletion timed out waiting for all nodes, retrying... [2022-03-01 04:30:17,843] - [rest_client:1078] ERROR - socket error while connecting to http://172.23.104.172:8091/pools/default/buckets/default error [Errno 111] Connection refused [2022-03-01 04:30:20,850] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/pools/default/buckets/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: status: 404, content: b'Requested resource not found.\r\n' b'Requested resource not found.\r\n' auth: Administrator:password [2022-03-01 04:30:20,851] - [rest_client:2879] WARNING - Bucket does not exist..exiting bucket delete [2022-03-01 04:30:20,861] - [bucket_helper:136] ERROR - Unable to get timings for bucket [2022-03-01 04:30:20,861] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 04:30:20,864] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 04:30:20,864] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 04:30:20,978] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.106.52 [2022-03-01 04:30:21,633] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 04:30:21,639] - [rest_client:140] INFO - node 172.23.106.52 existing buckets : [] [2022-03-01 04:30:21,639] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.106.52 [2022-03-01 04:30:21,646] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:30:21,653] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:30:21,660] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:30:21,667] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:30:21,673] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:30:21,681] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:30:21,696] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:30:21,723] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 04:30:21,726] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:30:21,781] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:30:21,785] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 04:30:31,800] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:30:41,815] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:31:01,843] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 04:31:01,843] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:31:11,879] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.105.162', 8091)] [2022-03-01 04:31:11,882] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:11,886] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:11,894] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:11,898] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:11,906] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:11,910] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:11,918] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:11,922] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:11,928] - [failoverbasetests:109] INFO - ============== tearDown was finished for test #10 test_failover_stop_server ============== [2022-03-01 04:31:11,941] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:31:11,941] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 0.9334006054490414, 'mem_free': 3035471872, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:31:11,941] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:31:11,944] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:12,082] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:31:12,358] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:12,739] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:12,741] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:12,879] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:31:13,157] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:13,544] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:13,547] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:13,651] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:31:13,958] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:14,289] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:14,292] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:14,396] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:31:14,698] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:15,035] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:15,037] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:15,141] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:31:15,426] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:15,765] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:15,767] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:15,872] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:31:16,189] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:16,524] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:28,974] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #10 test_failover_stop_server ============== [2022-03-01 04:31:28,998] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 04:31:29,007] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 04:31:29,014] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:31:29,020] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:31:29,025] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:31:29,031] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:31:29,031] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 04:31:29,043] - [rest_client:1073] ERROR - DELETE http://172.23.106.52:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 04:31:29,043] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 04:31:29,049] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,069] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 04:31:29,069] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,073] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 04:31:29,079] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 04:31:29,079] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,084] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 04:31:29,088] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 04:31:29,088] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,091] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 04:31:29,094] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 04:31:29,094] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,097] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 04:31:29,100] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 04:31:29,100] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,102] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 04:31:29,105] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 04:31:29,105] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,108] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 04:31:29,108] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #10 test_failover_stop_server ============== [2022-03-01 04:31:29,108] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 04:31:29,109] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 summary so far suite failover.failovertests.FailoverTests , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_10 ok ---------------------------------------------------------------------- Ran 1 test in 686.271s OK test_failover_then_add_back (failover.failovertests.FailoverTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_11 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t failover.failovertests.FailoverTests.test_failover_then_add_back,replicas=1,num_failed_nodes=1,items=10000,num_views=1,withViewsOps=True,createIndexesDuringFailover=True,upr_check=False,recoveryType=full,graceful=True,GROUP=P0;GRACEFUL Test Input params: {'replicas': '1', 'num_failed_nodes': '1', 'items': '10000', 'num_views': '1', 'withViewsOps': 'True', 'createIndexesDuringFailover': 'True', 'upr_check': 'False', 'recoveryType': 'full', 'graceful': 'True', 'GROUP': 'P0;GRACEFUL', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_11'} [2022-03-01 04:31:29,131] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:29,264] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:31:29,538] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:29,921] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:29,928] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:31:29,941] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 04:31:29,941] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #11 test_failover_then_add_back============== [2022-03-01 04:31:29,942] - [basetestcase:331] INFO - initializing cluster [2022-03-01 04:31:30,136] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:31:30,141] - [task:159] INFO - {'uptime': '101', 'memoryTotal': 3947372544, 'memoryFree': 3268612096, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:31:30,143] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:30,143] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 04:31:30,143] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 04:31:30,144] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 04:31:30,149] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 04:31:30,149] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 04:31:30,149] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 04:31:30,154] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 04:31:30,160] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 04:31:30,160] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 04:31:30,189] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:31:30,189] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 04:31:30,400] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:31:30,402] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:30,547] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:31:30,832] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:31,303] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:31,304] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:31:31,363] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:31:31,363] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:31:31,367] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:31,369] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:31,379] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:31:31,410] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:31:31,415] - [task:159] INFO - {'uptime': '4842', 'memoryTotal': 3947372544, 'memoryFree': 3040735232, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:31:31,425] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 04:31:31,439] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:31:31,439] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 04:31:31,572] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:31:31,574] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:31,712] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:31:31,992] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:32,380] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:32,381] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:31:32,461] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:31:32,461] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:31:32,466] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:32,468] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:32,476] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:31:32,502] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:31:32,505] - [task:159] INFO - {'uptime': '4817', 'memoryTotal': 4201627648, 'memoryFree': 3539091456, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:31:32,507] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:32,507] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:31:32,511] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:31:32,511] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 04:31:32,569] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:31:32,571] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:32,675] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:31:32,983] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:33,322] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:33,323] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:31:33,392] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:31:33,392] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:31:33,396] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:33,398] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:33,402] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:31:33,415] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:31:33,419] - [task:159] INFO - {'uptime': '4823', 'memoryTotal': 4201627648, 'memoryFree': 3561480192, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:31:33,421] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:33,421] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:31:33,425] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:31:33,425] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 04:31:33,482] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:31:33,484] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:33,588] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:31:33,906] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:34,252] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:34,253] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:31:34,323] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:31:34,325] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:31:34,328] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:34,330] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:34,334] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:31:34,347] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:31:34,350] - [task:159] INFO - {'uptime': '4823', 'memoryTotal': 4201627648, 'memoryFree': 3512934400, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:31:34,352] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:34,352] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:31:34,355] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:31:34,356] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 04:31:34,416] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:31:34,418] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:34,521] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:31:34,815] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:35,156] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:35,157] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:31:35,225] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:31:35,225] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:31:35,228] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:35,231] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:35,234] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:31:35,251] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:31:35,254] - [task:159] INFO - {'uptime': '4822', 'memoryTotal': 4201627648, 'memoryFree': 3518124032, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:31:35,256] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:31:35,256] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:31:35,259] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:31:35,260] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 04:31:35,317] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:31:35,319] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:31:35,424] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:31:35,728] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:31:36,057] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:31:36,058] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:31:36,125] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:31:36,125] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:31:36,128] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:36,130] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:31:36,134] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:31:36,144] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 04:31:36,160] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 04:31:36,161] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2022-03-01 04:31:36,305] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 04:31:41,311] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 04:31:41,325] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 04:31:42,150] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 04:31:42,160] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:31:52,177] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:31:52,177] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:32:08,939] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 04:32:08,950] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:32:18,966] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:32:18,966] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:32:32,244] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 04:32:32,257] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:32:42,264] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-03-01 04:32:42,264] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:32:57,445] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 04:32:57,467] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:33:07,484] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:33:07,484] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:33:20,954] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 04:33:20,981] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:33:30,998] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:33:30,998] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:33:44,810] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 04:33:44,811] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 04:33:44,811] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 04:33:44,811] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 04:33:44,811] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 04:33:44,822] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:33:44,880] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:33:54,894] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:33:54,905] - [task:902] INFO - rebalancing was completed with progress: 100% in 10.025316953659058 sec [2022-03-01 04:33:54,908] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:33:55,048] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:33:55,325] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:33:55,715] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:33:55,715] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:33:55,754] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:33:55,754] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:33:55,754] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 04:33:55,936] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 04:33:55,969] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 04:33:55,970] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 04:33:56,718] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:33:56,931] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:33:57,059] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:33:57,182] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:33:57,381] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:33:57,524] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:33:57,926] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:33:58,091] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:33:58,229] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:33:58,433] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:33:58,568] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:33:58,712] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:33:59,589] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 04:33:59,591] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #11 test_failover_then_add_back ============== [2022-03-01 04:33:59,603] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:33:59,739] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:34:00,004] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:34:00,385] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:34:00,388] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:34:00,526] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:34:00,808] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:34:01,201] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:34:01,204] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:34:01,310] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:34:01,626] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:34:01,955] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:34:01,958] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:34:02,065] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:34:02,376] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:34:02,714] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:34:02,717] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:34:02,818] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:34:03,108] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:34:03,446] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:34:03,449] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:34:03,555] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:34:03,869] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:34:04,205] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:34:16,434] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:34:16,434] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 11.18437737763124, 'mem_free': 3073081344, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:34:16,435] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.148373983739837, 'mem_free': 3416399872, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 04:34:16,435] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.888324873096447, 'mem_free': 3371728896, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 04:34:16,435] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.158134820070958, 'mem_free': 3400908800, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 04:34:16,435] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 6.045212090424181, 'mem_free': 3366576128, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 04:34:16,435] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 14.10682340914899, 'mem_free': 3069624320, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:34:16,435] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:34:16,435] - [basetestcase:559] INFO - Time to execute basesetup : 167.30641341209412 [2022-03-01 04:34:16,436] - [failoverbasetests:85] INFO - ============== FailoverBaseTest setup was finished for test #11 test_failover_then_add_back ============== [2022-03-01 04:34:16,436] - [failovertests:49] INFO - Picking node 172.23.104.172 as reference node for test case [2022-03-01 04:34:16,436] - [failovertests:319] INFO - num_replicas : 1 [2022-03-01 04:34:16,436] - [failovertests:320] INFO - recoveryType : ['full'] [2022-03-01 04:34:16,436] - [failovertests:321] INFO - failover_reason : normal [2022-03-01 04:34:16,436] - [failovertests:322] INFO - num_failed_nodes : 1 [2022-03-01 04:34:16,436] - [failovertests:323] INFO - picking server : ip:172.23.104.172 port:8091 ssh_username:nonroot as the master [2022-03-01 04:34:16,488] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:34:16,488] - [basetestcase:829] INFO - sleep for 120 secs. ... [2022-03-01 04:36:16,706] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:36:16,806] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:36:16,875] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:36:17,019] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:36:17,090] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:36:17,170] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:36:17,672] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:36:17,675] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:36:17,676] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:36:17,677] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:36:17,788] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:36:17,802] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:36:17,805] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:36:17,807] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:36:17,979] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:36:17,986] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:36:17,987] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:36:17,991] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:36:18,053] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:36:18,053] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:36:18,054] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:36:18,056] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:36:18,126] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:36:18,126] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:36:18,128] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:36:18,129] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:36:18,287] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:36:18,288] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:36:18,290] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:36:18,290] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:36:19,320] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:36:19,330] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:36:19,552] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:19,694] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:36:19,967] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:20,351] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:20,352] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:20,647] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:20,649] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:36:20,651] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:20,786] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:36:21,064] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:21,444] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:21,444] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:36:22,898] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:22,904] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:36:22,907] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:23,048] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:36:23,319] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:23,700] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:23,701] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:24,000] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:24,002] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:36:24,004] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:24,147] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:36:24,417] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:24,821] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:24,822] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:36:26,305] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:26,312] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:36:26,314] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:26,423] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:36:26,746] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:27,084] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:27,085] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:27,261] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:27,262] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:36:27,265] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:27,365] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:36:27,676] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:28,010] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:28,011] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:36:28,641] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:28,647] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:36:28,649] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:28,755] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:36:29,073] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:29,409] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:29,410] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:29,583] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:29,584] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:36:29,586] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:29,685] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:36:29,986] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:30,327] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:30,328] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:36:30,982] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:30,988] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:36:30,990] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:31,097] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:36:31,386] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:31,724] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:31,724] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:31,900] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:31,901] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:36:31,903] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:32,002] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:36:32,285] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:32,618] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:32,619] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:36:33,280] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:33,287] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:36:33,289] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:33,395] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:36:33,713] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:34,046] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:34,047] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:34,221] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:34,223] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:36:34,225] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:34,328] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:36:34,635] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:34,967] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:34,968] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:36:35,592] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:35,598] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:36:35,624] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:36:35,636] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:36:36,601] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:36,738] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:36:37,000] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:37,379] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:37,380] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:37,668] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:37,671] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:37,808] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:36:38,074] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:38,453] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:38,454] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:38,741] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:38,745] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:38,846] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:36:39,159] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:39,494] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:39,495] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:39,670] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:39,674] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:39,779] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:36:40,090] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:40,427] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:40,428] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:40,600] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:40,604] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:40,706] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:36:40,985] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:41,318] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:41,319] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:41,494] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:41,497] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:41,598] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:36:41,902] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:42,239] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:42,240] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:42,412] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:42,413] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:36:42,415] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:42,549] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:36:42,821] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:43,208] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:43,209] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:43,465] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:43,468] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:43,603] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:36:43,872] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:44,258] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:44,259] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:44,550] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:44,554] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:44,659] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:36:44,965] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:45,303] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:45,303] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:45,475] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:45,478] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:45,581] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:36:45,891] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:46,238] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:46,239] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:46,415] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:46,418] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:46,519] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:36:46,802] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:47,138] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:47,139] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:47,317] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:47,320] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:47,423] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:36:47,727] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:48,060] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:48,060] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:48,227] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:48,228] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:36:48,230] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:48,365] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:36:48,629] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:49,007] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:49,008] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:49,304] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:49,308] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:49,446] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:36:49,718] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:50,097] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:50,098] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:50,352] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:50,355] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:50,457] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:36:50,762] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:51,094] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:51,095] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:51,268] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:51,272] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:51,374] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:36:51,684] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:52,026] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:52,027] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:52,199] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:52,202] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:52,307] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:36:52,595] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:52,934] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:52,935] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:53,113] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:53,116] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:53,217] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:36:53,525] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:53,858] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:53,859] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:54,032] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:54,034] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:36:54,036] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:54,175] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:36:54,450] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:54,826] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:54,827] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:55,118] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:55,121] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:55,260] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:36:55,541] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:55,932] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:55,933] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:56,199] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:56,203] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:56,306] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:36:56,623] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:56,958] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:56,959] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:57,135] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:57,138] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:57,245] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:36:57,557] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:57,900] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:57,900] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:58,076] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:58,080] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:58,190] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:36:58,475] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:58,814] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:58,815] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:58,987] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:58,990] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:36:59,095] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:36:59,408] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:36:59,747] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:36:59,747] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:36:59,920] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:36:59,922] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:37:00,959] - [rest_client:1073] ERROR - GET http://172.23.104.172:8092//default/_design/dev_ddoc1 body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: not_found b'{"error":"not_found","reason":"missing"}\n' auth: Administrator:password [2022-03-01 04:37:00,960] - [rest_client:548] INFO - -->create_design_document [2022-03-01 04:37:01,019] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?stale=ok [2022-03-01 04:37:01,227] - [task:2865] INFO - view : default_view0 was created successfully in ddoc: dev_ddoc1 [2022-03-01 04:37:01,254] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:37:01,254] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:37:01,508] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:01,647] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:37:01,913] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:02,295] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:02,295] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:37:02,784] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:37:02,792] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:02,894] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:37:03,197] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:03,537] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:03,537] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:37:03,791] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:37:03,799] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:03,900] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:37:04,189] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:04,530] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:04,531] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:37:04,791] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:37:04,798] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:04,905] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:37:05,212] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:05,540] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:05,541] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:37:05,785] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:37:05,793] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:05,899] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:37:06,217] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:06,552] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:06,553] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:37:06,800] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:37:06,807] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:06,944] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:37:07,214] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:07,593] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:07,594] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:37:08,081] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:37:08,115] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:37:08,128] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:37:08,144] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:08,283] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:37:08,554] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:08,933] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:08,934] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:37:10,112] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:37:10,351] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:10,483] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:37:10,764] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:11,146] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:11,147] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:37:12,332] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:37:12,572] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:12,677] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:37:12,985] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:13,319] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:13,320] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:37:13,900] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:37:14,111] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:14,215] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:37:14,520] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:14,862] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:14,863] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:37:15,431] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:37:15,646] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:15,749] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:37:16,034] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:16,380] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:16,381] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:37:16,940] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:37:17,152] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:37:17,256] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:37:17,571] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:37:17,905] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:37:17,905] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:37:18,485] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:37:19,307] - [task:4078] INFO - Failing over 172.23.105.126:8091 with graceful=True [2022-03-01 04:37:19,386] - [rest_client:1790] INFO - fail_over node ns_1@172.23.105.126 successful [2022-03-01 04:37:19,386] - [task:4058] INFO - 0 seconds sleep after failover, for nodes to go pending.... [2022-03-01 04:37:19,394] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:37:37,930] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:37:38,741] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:37:38,742] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:37:39,754] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:37:39,762] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:37:39,769] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:37:39,775] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:37:39,781] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:37:39,784] - [rest_client:2012] INFO - rebalance percentage : 100.00 % [2022-03-01 04:37:49,798] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:37:49,798] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:37:59,828] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:37:59,828] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:37:59,828] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:38:00,042] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:00,182] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:38:00,449] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:00,830] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:00,831] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:38:01,312] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:01,320] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:01,423] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:38:01,706] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:02,039] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:02,040] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:38:02,285] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:02,293] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:02,396] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:38:02,704] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:03,032] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:03,033] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:38:03,280] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:03,288] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:03,394] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:38:03,706] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:04,036] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:04,037] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:38:04,290] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:04,298] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:04,433] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:38:04,714] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:05,102] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:05,103] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:38:05,574] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:05,602] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:38:05,602] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:38:05,612] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:38:05,613] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:38:05,814] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:05,951] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:38:06,223] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:06,602] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:06,603] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:38:06,905] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:06,906] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:38:06,909] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:07,049] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:38:07,324] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:07,709] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:07,709] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:38:08,905] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:08,910] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:38:08,912] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:09,048] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:38:09,326] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:09,713] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:09,714] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:38:09,972] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:09,974] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:38:09,977] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:10,117] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:38:10,389] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:10,772] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:10,774] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:38:12,016] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:12,022] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:38:12,024] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:12,149] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:38:12,459] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:12,795] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:12,796] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:38:12,970] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:12,971] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:38:12,973] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:13,075] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:38:13,389] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:13,719] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:13,719] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:38:14,264] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:14,269] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:38:14,271] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:14,379] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:38:14,669] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:15,009] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:15,009] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:38:15,191] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:15,192] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:38:15,195] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:15,294] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:38:15,572] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:15,905] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:15,906] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:38:16,444] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:16,450] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:38:16,452] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:16,555] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:38:16,858] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:17,194] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:17,194] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:38:17,372] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:17,373] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:38:17,375] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:17,480] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:38:17,790] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:18,122] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:18,122] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:38:18,626] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:18,631] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:38:18,633] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:38:18,736] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:38:19,045] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:38:19,390] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:38:19,391] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: echo 'check' > /home/nonroot/opt/couchbase/var/lib/couchbase/data/default/check.txt [2022-03-01 04:38:19,450] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:38:19,451] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 04:38:24,456] - [rest_client:1797] INFO - Going to set recoveryType=full for node :: ns_1@172.23.105.126 [2022-03-01 04:38:24,486] - [rest_client:1809] INFO - recoveryType for node ns_1@172.23.105.126 set successful [2022-03-01 04:38:24,486] - [basetestcase:829] INFO - sleep for 20 secs. After failover before invoking rebalance... ... [2022-03-01 04:38:44,507] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:38:44,563] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:38:44,669] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:38:45,504] - [rest_client:610] INFO - index query url: http://172.23.104.172:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=false [2022-03-01 04:38:46,418] - [task:3077] INFO - Server: 172.23.104.172, Design Doc: dev_ddoc1, View: default_view0, (10000 rows) expected, (10000 rows) returned [2022-03-01 04:38:46,418] - [task:3084] INFO - expected number of rows: '10000' was found for view query [2022-03-01 04:38:47,434] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:38:47,444] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:38:47,452] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:38:47,464] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:38:47,472] - [task:3740] INFO - task for monitoring indexer:_design/dev_ddoc1 completed [2022-03-01 04:38:47,475] - [rest_client:2012] INFO - rebalance percentage : 1.00 % [2022-03-01 04:38:57,492] - [rest_client:2012] INFO - rebalance percentage : 23.00 % [2022-03-01 04:39:17,513] - [rest_client:1951] INFO - rebalance progress took 30.04 seconds [2022-03-01 04:39:17,513] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:39:27,545] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:39:27,557] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:39:28,519] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:28,659] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:39:28,934] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:29,324] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:29,325] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:29,596] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:29,597] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:39:29,600] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:29,744] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:39:30,015] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:30,394] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:30,395] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:31,865] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:31,871] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:39:31,874] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:32,013] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:39:32,289] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:32,673] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:32,674] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:32,977] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:32,978] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:39:32,980] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:33,112] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:39:33,384] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:33,765] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:33,766] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:35,240] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:35,246] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:39:35,248] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:35,360] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:39:35,675] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:36,006] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:36,007] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:36,176] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:36,178] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:39:36,180] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:36,277] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:39:36,585] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:36,920] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:36,920] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:37,563] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:37,569] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:39:37,571] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:37,676] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:39:37,989] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:38,330] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:38,331] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:38,505] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:38,506] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:39:38,508] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:38,610] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:39:38,924] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:39,263] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:39,264] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:39,920] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:39,926] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:39:39,928] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:40,033] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:39:40,320] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:40,658] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:40,659] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:40,831] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:40,833] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:39:40,835] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:40,935] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:39:41,214] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:41,547] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:41,548] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:42,189] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:42,196] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:39:42,198] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:42,303] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:39:42,614] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:42,941] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:42,941] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:43,108] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:43,110] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:39:43,112] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:43,209] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:39:43,514] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:43,846] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:43,847] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:44,487] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:44,494] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:39:44,494] - [failovertests:285] INFO - Begin VERIFICATION for Add-back and rebalance [2022-03-01 04:39:44,517] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:39:44,527] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:39:44,537] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:39:44,546] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:39:45,497] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:45,635] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:39:45,913] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:46,293] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:46,294] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:46,593] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:46,594] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:39:46,596] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:46,731] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:39:47,000] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:47,376] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:47,378] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:48,861] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:48,867] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:39:48,869] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:49,003] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:39:49,276] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:49,660] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:49,661] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:49,956] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:49,957] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:39:49,959] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:50,102] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:39:50,373] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:50,757] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:50,758] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:52,255] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:52,262] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:39:52,264] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:52,364] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:39:52,679] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:53,012] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:53,013] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:53,153] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:53,154] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:39:53,156] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:53,260] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:39:53,569] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:53,905] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:53,906] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:54,570] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:54,577] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:39:54,579] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:54,684] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:39:54,990] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:55,323] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:55,323] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:55,496] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:55,497] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:39:55,499] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:55,597] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:39:55,896] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:56,244] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:56,245] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:56,891] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:56,899] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:39:56,901] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:57,001] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:39:57,279] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:57,615] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:57,616] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:39:57,784] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:57,785] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:39:57,787] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:57,884] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:39:58,171] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:58,511] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:58,512] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:39:59,152] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:39:59,159] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:39:59,161] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:39:59,265] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:39:59,579] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:39:59,917] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:39:59,917] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:00,091] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:00,092] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:40:00,094] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:00,197] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:40:00,514] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:00,851] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:00,851] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:40:01,493] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:01,499] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:40:01,666] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:40:01,777] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:40:01,857] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:40:02,008] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:40:02,077] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:40:02,153] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:40:02,391] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:40:02,501] - [task:2328] INFO - 0 items were verified [2022-03-01 04:40:06,127] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:40:06,127] - [task:2332] INFO - 10000 items were verified in 3.7355844974517822 sec.the average number of ops - 2676.0 per second [2022-03-01 04:40:06,151] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:40:06,163] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:40:06,531] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:06,671] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:40:06,950] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:07,336] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:07,337] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:07,599] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:07,602] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:07,747] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:40:08,025] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:08,403] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:08,404] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:08,702] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:08,705] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:08,811] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:40:09,129] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:09,468] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:09,469] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:09,645] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:09,648] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:09,750] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:40:10,063] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:10,403] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:10,403] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:10,579] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:10,581] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:10,685] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:40:10,980] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:11,318] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:11,318] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:11,495] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:11,498] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:11,604] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:40:11,909] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:12,238] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:12,239] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:12,408] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:12,409] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:40:12,412] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:12,549] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:40:12,827] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:13,217] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:13,218] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:13,518] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:13,521] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:13,655] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:40:13,946] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:14,326] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:14,327] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:14,615] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:14,618] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:14,720] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:40:15,033] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:15,369] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:15,370] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:15,546] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:15,549] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:15,651] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:40:15,958] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:16,295] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:16,296] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:16,470] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:16,473] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:16,573] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:40:16,857] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:17,191] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:17,191] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:17,365] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:17,368] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:17,471] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:40:17,782] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:18,114] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:18,115] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:18,296] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:18,297] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:40:18,299] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:18,441] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:40:18,732] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:19,117] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:19,118] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:19,377] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:19,380] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:19,517] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:40:19,789] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:20,176] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:20,177] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:20,427] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:20,431] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:20,532] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:40:20,838] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:21,175] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:21,175] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:21,350] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:21,353] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:21,457] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:40:21,771] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:22,109] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:22,110] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:22,284] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:22,287] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:22,389] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:40:22,673] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:23,010] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:23,011] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:23,145] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:23,148] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:23,246] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:40:23,558] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:23,894] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:23,895] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:24,033] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:24,034] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:40:24,036] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:24,174] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:40:24,453] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:24,855] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:24,856] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:25,153] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:25,157] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:25,290] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:40:25,619] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:25,996] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:25,998] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:26,304] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:26,307] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:26,408] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:40:26,721] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:27,062] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:27,062] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:27,229] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:27,232] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:27,336] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:40:27,640] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:27,980] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:27,981] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:28,157] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:28,160] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:28,265] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:40:28,547] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:28,883] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:28,884] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:29,057] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:29,060] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:29,165] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:40:29,473] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:29,809] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:29,810] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:40:29,981] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:40:29,982] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:40:29,983] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:40:31,530] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 1667 [2022-03-01 04:40:31,530] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 04:40:31,530] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 171 [2022-03-01 04:40:31,530] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 171 [2022-03-01 04:40:31,530] - [rebalance_helper:370] INFO - curr_items from 172.23.105.126:8091 : 1672 [2022-03-01 04:40:31,530] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.126:8091 : 0 [2022-03-01 04:40:31,531] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.126:8091 : 171 [2022-03-01 04:40:31,531] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.126:8091 : 171 [2022-03-01 04:40:31,531] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 1676 [2022-03-01 04:40:31,531] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 04:40:31,531] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 171 [2022-03-01 04:40:31,531] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 171 [2022-03-01 04:40:31,531] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 1658 [2022-03-01 04:40:31,531] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 04:40:31,531] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 171 [2022-03-01 04:40:31,532] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 171 [2022-03-01 04:40:31,532] - [rebalance_helper:370] INFO - curr_items from 172.23.105.162:8091 : 1671 [2022-03-01 04:40:31,532] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.162:8091 : 0 [2022-03-01 04:40:31,532] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.162:8091 : 170 [2022-03-01 04:40:31,532] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.162:8091 : 170 [2022-03-01 04:40:31,532] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 1656 [2022-03-01 04:40:31,532] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 04:40:31,532] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 170 [2022-03-01 04:40:31,533] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 170 [2022-03-01 04:40:31,533] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 04:40:31,533] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:40:31,980] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 04:40:31,981] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:40:31,984] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:40:32,086] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:40:32,387] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:40:32,723] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:40:32,797] - [basetestcase:829] INFO - sleep for 60 secs. ... [2022-03-01 04:41:32,857] - [basetestcase:2135] INFO - Begin Verification for data comparison [2022-03-01 04:41:32,880] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:41:32,890] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:41:32,903] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:33,041] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:41:33,320] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:33,721] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:33,722] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:41:34,900] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:41:35,148] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:35,288] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:41:35,556] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:35,936] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:35,937] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:41:37,114] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:41:37,363] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:37,469] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:41:37,789] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:38,128] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:38,129] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:41:38,710] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:41:38,919] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:39,025] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:41:39,341] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:39,687] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:39,687] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:41:40,268] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:41:40,480] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:40,587] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:41:40,879] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:41,219] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:41,220] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:41:41,792] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:41:42,006] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:42,112] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:41:42,427] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:42,757] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:42,758] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:41:43,329] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:41:43,604] - [basetestcase:2144] INFO - End Verification for data comparison [2022-03-01 04:41:43,658] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 04:41:43,669] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:41:43,680] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:41:43,691] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:41:43,700] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:41:43,702] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:43,842] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:41:44,115] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:44,493] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:44,494] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:44,782] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:44,783] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:45,059] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:45,062] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:45,165] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:41:45,467] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:45,807] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:45,808] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:45,975] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:45,976] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:46,145] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:46,148] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:46,246] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:41:46,526] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:46,860] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:46,861] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:47,023] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:47,024] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:47,186] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:47,189] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:47,294] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:41:47,603] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:47,942] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:47,943] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:48,113] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:48,114] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:48,246] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:48,249] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:48,356] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:41:48,673] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:49,005] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:49,006] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:49,169] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:49,170] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:49,334] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:49,337] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:49,472] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:41:49,749] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:50,134] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:50,135] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:50,413] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:50,414] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:41:50,710] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:50,711] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 04:41:50,767] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 04:41:50,767] - [failovertests:315] INFO - End VERIFICATION for Add-back and rebalance [2022-03-01 04:41:50,779] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:41:50,779] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:41:51,039] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:51,177] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:41:51,449] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:51,830] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:51,831] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:41:52,307] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:52,316] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:52,419] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:41:52,720] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:53,058] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:53,059] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:41:53,312] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:53,319] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:53,422] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:41:53,713] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:54,059] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:54,060] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:41:54,304] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:54,313] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:54,417] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:41:54,725] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:55,060] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:55,061] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:41:55,305] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:55,314] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:55,415] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:41:55,719] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:56,048] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:56,048] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:41:56,289] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:56,298] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:56,442] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:41:56,751] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:57,156] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:57,157] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:41:57,637] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:57,648] - [failoverbasetests:98] INFO - ============== tearDown was started for test #11 test_failover_then_add_back ============== [2022-03-01 04:41:57,650] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:57,793] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:41:58,066] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:41:58,447] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:41:58,448] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:41:58,528] - [remote_util:177] INFO - process beam.smp is running on 172.23.104.172: with pid 130199 [2022-03-01 04:41:58,528] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2022-03-01 04:41:58,671] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2022-03-01 04:41:58,671] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:41:58,671] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -F [2022-03-01 04:41:58,848] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:58,849] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -t nat -F [2022-03-01 04:41:58,910] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:41:58,910] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:59,047] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:41:59,047] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:41:59,047] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT memcached [2022-03-01 04:41:59,249] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:41:59,249] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:41:59,249] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT beam [2022-03-01 04:41:59,288] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:41:59,288] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:41:59,290] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:41:59,425] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:41:59,691] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:42:00,078] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:42:00,079] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:42:00,163] - [remote_util:177] INFO - process beam.smp is running on 172.23.106.52: with pid 27929 [2022-03-01 04:42:00,163] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 [2022-03-01 04:42:00,296] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as root [2022-03-01 04:42:00,296] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:42:00,297] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -F [2022-03-01 04:42:00,475] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:00,475] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -t nat -F [2022-03-01 04:42:00,538] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:00,539] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:00,673] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:42:00,673] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:42:00,673] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT memcached [2022-03-01 04:42:00,883] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:42:00,883] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:42:00,883] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT beam [2022-03-01 04:42:00,965] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:42:00,966] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:42:00,968] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:01,075] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:42:01,397] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:42:01,733] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:42:01,734] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:42:01,823] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.158: with pid 25310 [2022-03-01 04:42:01,824] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 [2022-03-01 04:42:01,925] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as root [2022-03-01 04:42:01,925] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:42:01,925] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -F [2022-03-01 04:42:02,157] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:02,157] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -t nat -F [2022-03-01 04:42:02,212] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:02,213] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:02,314] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:42:02,314] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:42:02,314] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT memcached [2022-03-01 04:42:02,557] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:42:02,557] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:42:02,558] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT beam [2022-03-01 04:42:02,620] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:42:02,620] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:42:02,622] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:02,730] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:42:03,049] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:42:03,389] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:42:03,390] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:42:03,476] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.126: with pid 10785 [2022-03-01 04:42:03,477] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 [2022-03-01 04:42:03,574] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as root [2022-03-01 04:42:03,574] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:42:03,575] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -F [2022-03-01 04:42:03,785] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:03,785] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -t nat -F [2022-03-01 04:42:03,841] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:03,841] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:03,950] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:42:03,950] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:42:03,951] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT memcached [2022-03-01 04:42:04,184] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:42:04,185] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:42:04,185] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT beam [2022-03-01 04:42:04,247] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:42:04,247] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:42:04,250] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:04,354] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:42:04,639] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:42:04,972] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:42:04,973] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:42:05,058] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.127: with pid 31854 [2022-03-01 04:42:05,058] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 [2022-03-01 04:42:05,157] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as root [2022-03-01 04:42:05,157] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:42:05,157] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -F [2022-03-01 04:42:05,350] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:05,351] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -t nat -F [2022-03-01 04:42:05,406] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:05,407] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:05,508] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:42:05,508] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:42:05,509] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT memcached [2022-03-01 04:42:05,727] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:42:05,727] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:42:05,727] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT beam [2022-03-01 04:42:05,793] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:42:05,793] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:42:05,795] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:05,902] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:42:06,209] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:42:06,542] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:42:06,543] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 04:42:06,629] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.162: with pid 10888 [2022-03-01 04:42:06,629] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 [2022-03-01 04:42:06,730] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as root [2022-03-01 04:42:06,730] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 04:42:06,730] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -F [2022-03-01 04:42:06,955] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:06,955] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -t nat -F [2022-03-01 04:42:07,010] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:42:07,011] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:42:07,111] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:42:07,111] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 04:42:07,112] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT memcached [2022-03-01 04:42:07,346] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 04:42:07,347] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 04:42:07,347] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT beam [2022-03-01 04:42:07,407] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 04:42:07,407] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 04:42:17,447] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 04:42:18,851] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 04:42:18,856] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 04:42:18,857] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 04:42:18,868] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 04:42:18,875] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:42:18,883] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:42:18,891] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:42:18,897] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:42:18,907] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:42:18,937] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 04:42:18,940] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:42:19,011] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:42:19,013] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 04:42:29,028] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:42:39,043] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 04:42:59,072] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 04:42:59,072] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:43:09,113] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.126', 8091), ('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.105.162', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 04:43:09,117] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:09,122] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:09,132] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:09,136] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:09,145] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:09,149] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:09,158] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:09,164] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:09,172] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:09,176] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:09,181] - [failoverbasetests:109] INFO - ============== tearDown was finished for test #11 test_failover_then_add_back ============== [2022-03-01 04:43:09,194] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:43:09,194] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.214267644826714, 'mem_free': 3018391552, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:43:09,194] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:43:09,196] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:09,330] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:43:09,601] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:09,979] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:09,981] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:10,115] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:43:10,391] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:10,786] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:10,788] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:10,895] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:43:11,206] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:11,532] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:11,535] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:11,639] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:43:11,945] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:12,281] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:12,285] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:12,390] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:43:12,685] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:13,021] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:13,025] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:13,130] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:43:13,445] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:13,788] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:25,958] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #11 test_failover_then_add_back ============== [2022-03-01 04:43:25,983] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 04:43:25,991] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 04:43:25,997] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 04:43:26,003] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 04:43:26,009] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 04:43:26,014] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 04:43:26,014] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 04:43:26,026] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 04:43:26,026] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 04:43:26,031] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,053] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 04:43:26,053] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,058] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 04:43:26,062] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 04:43:26,063] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,066] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 04:43:26,070] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 04:43:26,070] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,073] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 04:43:26,076] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 04:43:26,076] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,079] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 04:43:26,082] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 04:43:26,082] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,086] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 04:43:26,089] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 04:43:26,089] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,092] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 04:43:26,093] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #11 test_failover_then_add_back ============== [2022-03-01 04:43:26,093] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 04:43:26,093] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 summary so far suite failover.failovertests.FailoverTests , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_11 ok ---------------------------------------------------------------------- Ran 1 test in 716.973s OK test_failover_normal (failover.failovertests.FailoverTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_12 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t failover.failovertests.FailoverTests.test_failover_normal,replicas=1,num_failed_nodes=1,items=10000,vbuckets=1024,total_vbuckets=1024,stopGracefulFailover=True,graceful=True,GROUP=P0;GRACEFUL Test Input params: {'replicas': '1', 'num_failed_nodes': '1', 'items': '10000', 'vbuckets': '1024', 'total_vbuckets': '1024', 'stopGracefulFailover': 'True', 'graceful': 'True', 'GROUP': 'P0;GRACEFUL', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 12, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_12'} [2022-03-01 04:43:26,118] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:26,255] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:43:26,519] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:26,899] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:26,907] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 04:43:26,924] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 04:43:26,924] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #12 test_failover_normal============== [2022-03-01 04:43:26,925] - [basetestcase:331] INFO - initializing cluster [2022-03-01 04:43:27,123] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:43:27,128] - [task:159] INFO - {'uptime': '816', 'memoryTotal': 3947372544, 'memoryFree': 3028094976, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:43:27,134] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 04:43:27,134] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 04:43:27,134] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 04:43:27,144] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 04:43:27,145] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 04:43:27,145] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 04:43:27,154] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 04:43:27,162] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 04:43:27,162] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 04:43:27,165] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 04:43:27,165] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 04:43:27,165] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:43:27,165] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 04:43:27,290] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:43:27,292] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:27,429] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:43:27,698] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:28,083] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:28,084] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:43:28,161] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:43:28,162] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:43:28,165] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:28,168] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:28,174] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:43:28,200] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:43:28,205] - [task:159] INFO - {'uptime': '5558', 'memoryTotal': 3947372544, 'memoryFree': 3220332544, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 04:43:28,207] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:28,207] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 04:43:28,212] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:43:28,212] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 04:43:28,348] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:43:28,350] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:28,495] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:43:28,790] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:29,205] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:29,206] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:43:29,249] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:43:29,249] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:43:29,253] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:29,256] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:29,261] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:43:29,278] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:43:29,280] - [task:159] INFO - {'uptime': '5534', 'memoryTotal': 4201627648, 'memoryFree': 3535376384, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:43:29,282] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:29,282] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:43:29,285] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:43:29,286] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 04:43:29,343] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:43:29,346] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:29,450] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:43:29,762] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:30,097] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:30,098] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:43:30,166] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:43:30,166] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:43:30,169] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:30,171] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:30,176] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:43:30,190] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:43:30,193] - [task:159] INFO - {'uptime': '5540', 'memoryTotal': 4201627648, 'memoryFree': 3532214272, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:43:30,195] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:30,195] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:43:30,198] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:43:30,199] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 04:43:30,258] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:43:30,260] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:30,361] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:43:30,665] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:31,000] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:31,000] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:43:31,071] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:43:31,071] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:43:31,075] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:31,077] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:31,081] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:43:31,093] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:43:31,096] - [task:159] INFO - {'uptime': '5540', 'memoryTotal': 4201627648, 'memoryFree': 3507322880, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:43:31,098] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:31,098] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:43:31,101] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:43:31,101] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 04:43:31,159] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:43:31,161] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:31,268] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:43:31,560] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:31,890] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:31,891] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:43:31,961] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:43:31,961] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:43:31,964] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:31,967] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:31,970] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:43:31,986] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 04:43:31,990] - [task:159] INFO - {'uptime': '5540', 'memoryTotal': 4201627648, 'memoryFree': 3514753024, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 04:43:31,992] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 04:43:31,992] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 04:43:31,995] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 04:43:31,996] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 04:43:32,053] - [rest_client:1151] INFO - --> status:True [2022-03-01 04:43:32,055] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:43:32,162] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:43:32,478] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:43:32,809] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:43:32,810] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:43:32,878] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:43:32,878] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:43:32,881] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:32,883] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 04:43:32,887] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 04:43:32,901] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 04:43:33,074] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 04:43:38,080] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 04:43:38,096] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 04:43:38,908] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 04:43:38,917] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:43:48,933] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:43:48,933] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:44:05,067] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 04:44:05,080] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:44:15,097] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:44:15,097] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:44:28,673] - [task:773] INFO - adding node 172.23.105.126:8091 to cluster [2022-03-01 04:44:28,691] - [rest_client:1613] INFO - adding remote node @172.23.105.126:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:44:38,708] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:44:38,708] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:44:52,215] - [task:773] INFO - adding node 172.23.105.127:8091 to cluster [2022-03-01 04:44:52,249] - [rest_client:1613] INFO - adding remote node @172.23.105.127:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:45:02,265] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 04:45:02,265] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:45:16,005] - [task:773] INFO - adding node 172.23.105.162:8091 to cluster [2022-03-01 04:45:16,022] - [rest_client:1613] INFO - adding remote node @172.23.105.162:18091 to this cluster @172.23.104.172:8091 [2022-03-01 04:45:26,029] - [rest_client:1951] INFO - rebalance progress took 10.01 seconds [2022-03-01 04:45:26,029] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:45:39,823] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveAdded [2022-03-01 04:45:39,823] - [rest_client:2621] INFO - Node 172.23.105.127 not part of cluster inactiveAdded [2022-03-01 04:45:39,823] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 04:45:39,823] - [rest_client:2621] INFO - Node 172.23.105.162 not part of cluster inactiveAdded [2022-03-01 04:45:39,823] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 04:45:39,836] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:45:39,894] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:45:49,909] - [rest_client:2012] INFO - rebalance percentage : 27.00 % [2022-03-01 04:45:49,909] - [task:843] INFO - Rebalance - status: running, progress: 27.00% [2022-03-01 04:46:09,948] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 04:46:09,960] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.065502166748047 sec [2022-03-01 04:46:09,962] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:10,097] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:46:10,372] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:10,753] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:10,754] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 04:46:10,833] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:46:10,833] - [remote_util:5219] INFO - ['ok'] [2022-03-01 04:46:10,834] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 04:46:10,995] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 04:46:11,031] - [rest_client:3014] INFO - 0.04 seconds to create bucket default [2022-03-01 04:46:11,031] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 04:46:11,905] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:46:12,055] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:46:12,187] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:46:12,412] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:46:12,547] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:46:12,703] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:46:13,087] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:46:13,253] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:46:13,502] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:46:13,626] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:46:13,756] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:46:13,981] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:46:14,892] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 04:46:14,894] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #12 test_failover_normal ============== [2022-03-01 04:46:14,905] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:15,040] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:46:15,314] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:15,689] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:15,692] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:15,830] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:46:16,106] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:16,491] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:16,494] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:16,600] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:46:16,919] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:17,253] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:17,256] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:17,362] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:46:17,668] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:18,006] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:18,009] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:18,111] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:46:18,400] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:18,735] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:18,737] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:46:18,842] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:46:19,157] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:46:19,483] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:46:31,761] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 04:46:31,761] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 21.61115414407436, 'mem_free': 2954530816, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 04:46:31,761] - [basetestcase:585] INFO - 172.23.105.126:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.839353372063652, 'mem_free': 3368660992, 'mem_total': 4201627648, 'swap_mem_used': 52428800, 'swap_mem_total': 3758092288} [2022-03-01 04:46:31,761] - [basetestcase:585] INFO - 172.23.105.127:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 2.145922746781116, 'mem_free': 3332530176, 'mem_total': 4201627648, 'swap_mem_used': 59703296, 'swap_mem_total': 3758092288} [2022-03-01 04:46:31,761] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.497846465670129, 'mem_free': 3371606016, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 04:46:31,761] - [basetestcase:585] INFO - 172.23.105.162:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.9691996970462, 'mem_free': 3352539136, 'mem_total': 4201627648, 'swap_mem_used': 112197632, 'swap_mem_total': 3758092288} [2022-03-01 04:46:31,761] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 8.945767866193615, 'mem_free': 3063721984, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 04:46:31,762] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 04:46:31,762] - [basetestcase:559] INFO - Time to execute basesetup : 185.64664721488953 [2022-03-01 04:46:31,762] - [failoverbasetests:85] INFO - ============== FailoverBaseTest setup was finished for test #12 test_failover_normal ============== [2022-03-01 04:46:31,762] - [failovertests:49] INFO - Picking node 172.23.104.172 as reference node for test case [2022-03-01 04:46:31,762] - [failovertests:319] INFO - num_replicas : 1 [2022-03-01 04:46:31,762] - [failovertests:320] INFO - recoveryType : ['delta'] [2022-03-01 04:46:31,762] - [failovertests:321] INFO - failover_reason : normal [2022-03-01 04:46:31,763] - [failovertests:322] INFO - num_failed_nodes : 1 [2022-03-01 04:46:31,763] - [failovertests:323] INFO - picking server : ip:172.23.104.172 port:8091 ssh_username:nonroot as the master [2022-03-01 04:46:31,817] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:46:31,817] - [basetestcase:829] INFO - sleep for 120 secs. ... [2022-03-01 04:48:32,028] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:48:32,137] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:48:32,306] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:48:32,379] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:48:32,457] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:48:32,627] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:48:33,015] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:48:33,015] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:48:33,015] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:48:33,016] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:48:33,253] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:48:33,261] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:48:33,263] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:48:33,266] - [data_helper:308] INFO - creating direct client 172.23.105.126:11210 default [2022-03-01 04:48:33,335] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:48:33,336] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:48:33,337] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:48:33,345] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:48:33,410] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:48:33,411] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:48:33,414] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:48:33,416] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:48:33,579] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:48:33,580] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:48:33,580] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:48:33,581] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:48:33,670] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:48:33,671] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:48:33,672] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:48:33,673] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:48:34,712] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:48:34,723] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:48:34,879] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:35,021] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:48:35,296] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:35,673] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:35,674] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:35,971] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:35,973] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:48:35,975] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:36,120] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:48:36,394] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:36,772] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:36,773] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:48:38,239] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:38,245] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:48:38,247] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:38,386] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:48:38,654] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:39,038] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:39,039] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:39,296] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:39,298] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:48:39,300] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:39,434] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:48:39,704] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:40,089] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:40,090] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:48:41,557] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:41,564] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:48:41,566] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:41,676] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:48:41,997] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:42,329] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:42,330] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:42,509] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:42,511] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:48:42,513] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:42,616] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:48:42,917] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:43,249] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:43,250] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:48:43,888] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:43,894] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:48:43,896] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:44,001] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:48:44,316] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:44,650] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:44,651] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:44,831] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:44,832] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:48:44,834] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:44,938] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:48:45,243] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:45,583] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:45,583] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:48:46,217] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:46,224] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.126:8091',default bucket [2022-03-01 04:48:46,226] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:46,329] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:48:46,622] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:46,956] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:46,958] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:47,144] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:47,146] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:48:47,150] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:47,251] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:48:47,534] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:47,866] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:47,867] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:48:48,506] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:48,512] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:48:48,514] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:48,621] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:48:48,998] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:49,371] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:49,372] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:49,538] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:49,540] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:48:49,543] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:49,645] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:48:49,957] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:50,293] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:50,294] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:48:50,940] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:50,948] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:48:50,971] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:48:50,982] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:48:51,951] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:52,088] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:48:52,367] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:52,746] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:52,747] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:53,048] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:53,053] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:53,188] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:48:53,459] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:53,839] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:53,840] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:54,133] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:54,136] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:54,241] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:48:54,562] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:54,901] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:54,902] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:55,034] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:55,036] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:55,141] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:48:55,439] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:55,776] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:55,776] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:55,949] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:55,952] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:56,056] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:48:56,339] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:56,669] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:56,670] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:56,844] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:56,847] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:56,954] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:48:57,265] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:57,596] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:57,597] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:57,765] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:57,766] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:48:57,769] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:57,916] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:48:58,184] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:58,564] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:58,565] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:48:58,855] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:48:58,859] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:48:58,993] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:48:59,375] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:48:59,760] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:48:59,761] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:00,064] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:00,070] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:00,172] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:00,485] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:00,814] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:00,814] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:00,989] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:00,992] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:01,096] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:01,397] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:01,732] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:01,732] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:01,905] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:01,908] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:02,011] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:02,293] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:02,628] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:02,629] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:02,805] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:02,808] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:02,911] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:03,215] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:03,554] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:03,555] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:03,729] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:03,730] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:49:03,732] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:03,868] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:04,133] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:04,510] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:04,511] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:04,799] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:04,802] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:04,936] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:05,227] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:05,616] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:05,618] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:05,918] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:05,921] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:06,025] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:06,373] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:06,726] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:06,727] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:06,902] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:06,905] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:07,007] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:07,311] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:07,654] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:07,655] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:07,827] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:07,831] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:07,932] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:08,216] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:08,551] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:08,552] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:08,727] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:08,730] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:08,834] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:09,144] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:09,480] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:09,481] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:09,656] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:09,657] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:49:09,660] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:09,803] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:10,072] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:10,454] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:10,455] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:10,769] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:10,772] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:10,910] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:11,183] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:11,565] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:11,566] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:11,869] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:11,872] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:11,974] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:12,288] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:12,620] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:12,621] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:12,793] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:12,797] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:12,902] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:13,203] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:13,541] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:13,542] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:13,718] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:13,721] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:13,830] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:14,122] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:14,468] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:14,469] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:14,648] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:14,651] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:14,755] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:15,078] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:15,420] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:15,420] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:49:15,602] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:15,603] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.126:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:49:15,641] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:49:15,642] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:49:15,806] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:15,944] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:16,228] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:16,605] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:16,607] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:17,074] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:17,082] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:17,186] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:17,494] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:17,828] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:17,828] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:18,078] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:18,086] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:18,190] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:18,477] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:18,815] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:18,816] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:19,062] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:19,071] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:19,172] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:19,483] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:19,817] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:19,818] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:20,063] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:20,071] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:20,174] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:20,475] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:20,808] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:20,809] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:21,060] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:21,068] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:21,213] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:21,504] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:21,889] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:21,890] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:22,358] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:22,392] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:49:22,402] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:49:22,416] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:22,551] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:22,819] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:23,204] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:23,205] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:49:24,380] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:49:24,616] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:24,752] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:25,024] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:25,408] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:25,409] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:49:26,603] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:49:26,852] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:26,956] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:27,265] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:27,597] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:27,598] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:49:28,161] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:49:28,371] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:28,477] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:28,789] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:29,134] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:29,135] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:49:29,709] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:49:29,925] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:30,027] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:30,306] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:30,644] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:30,645] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:49:31,226] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:49:31,430] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:31,536] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:31,843] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:32,175] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:32,176] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:49:32,749] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:49:32,998] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:33,138] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:33,418] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:33,805] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:33,806] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:34,233] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:34,241] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:34,377] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:34,647] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:35,031] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:35,032] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:35,510] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:35,518] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:35,618] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:35,929] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:36,264] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:36,265] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:36,508] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:36,516] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:36,619] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:36,926] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:37,268] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:37,269] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:37,517] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:37,524] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:37,625] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:37,909] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:38,247] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:38,247] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:38,500] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:38,508] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:38,612] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:38,922] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:39,254] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:39,254] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:49:39,500] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:39,561] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:39,698] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:39,967] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:40,351] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:40,352] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:40,593] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:40,594] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:40,831] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:40,834] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:40,967] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:41,241] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:41,625] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:41,626] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:41,901] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:41,902] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:42,174] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:42,178] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:42,281] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:42,591] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:42,933] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:42,934] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:43,104] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:43,105] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:43,230] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:43,233] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:43,339] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:43,645] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:43,993] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:43,994] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:44,156] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:44,157] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:44,322] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:44,325] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:44,430] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:44,725] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:45,067] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:45,068] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:45,237] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:45,238] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:45,362] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:45,365] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:45,473] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:45,781] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:46,117] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:46,118] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:46,281] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:46,282] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:49:46,408] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:46,411] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:46,550] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:49:46,825] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:47,203] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:47,203] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:49:47,507] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:47,607] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:47,746] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:49:48,026] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:48,407] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:48,408] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:49:48,781] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:48,787] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:48,893] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:49:49,207] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:49,540] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:49,540] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:49:49,736] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:49,743] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:49,848] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 04:49:50,162] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:50,505] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:50,505] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:49:50,663] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:50,670] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:50,774] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:49:51,057] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:51,396] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:51,397] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:49:51,590] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:51,596] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:49:51,703] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:49:52,009] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:49:52,348] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:49:52,348] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:49:52,506] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:49:52,704] - [rest_client:1790] INFO - fail_over node ns_1@172.23.105.126 successful [2022-03-01 04:49:52,705] - [failovertests:692] INFO - Stopping Graceful Failover [2022-03-01 04:49:52,710] - [rest_client:3190] WARNING - rebalance is not stopped yet after 1 sec [2022-03-01 04:49:53,714] - [rest_client:3194] INFO - rebalance was stopped [2022-03-01 04:49:53,715] - [basetestcase:829] INFO - sleep for 120 secs. ... [2022-03-01 04:51:53,779] - [failovertests:376] INFO - Start Graceful Failover Again ! [2022-03-01 04:51:53,779] - [basetestcase:829] INFO - sleep for 120 secs. ... [2022-03-01 04:53:53,933] - [rest_client:1790] INFO - fail_over node ns_1@172.23.105.126 successful [2022-03-01 04:53:53,933] - [basetestcase:829] INFO - sleep for 180 secs. ... [2022-03-01 04:56:54,015] - [failovertests:382] INFO - chosen: 1 get_failover_count: 1 [2022-03-01 04:56:57,030] - [rest_client:99] INFO - rebalance reached >100% in 3.004870891571045 seconds [2022-03-01 04:56:57,031] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 04:56:57,050] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:56:57,050] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:56:57,060] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:56:57,061] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:56:57,070] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:56:57,070] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:56:57,080] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:56:57,080] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:56:57,082] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:56:57,217] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:56:57,505] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:56:57,896] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:56:57,897] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:56:58,178] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:56:58,179] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:56:58,423] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:56:58,426] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:56:58,568] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:56:58,850] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:56:59,230] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:56:59,231] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:56:59,509] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:56:59,510] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:56:59,752] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:56:59,754] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:56:59,866] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:57:00,202] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:00,536] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:00,536] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:57:00,706] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:00,707] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:57:00,869] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:00,872] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:00,975] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:57:01,267] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:01,608] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:01,609] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:57:01,773] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:01,774] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:57:01,903] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:01,906] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:02,015] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:57:02,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:02,666] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:02,667] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:57:02,834] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:02,835] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:57:02,995] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:02,996] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 04:57:02,996] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 04:57:03,018] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:57:03,018] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:57:03,018] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 04:57:03,148] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:03,291] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:57:03,564] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:03,956] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:03,956] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:57:04,429] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:04,437] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:04,539] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:57:04,821] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:05,157] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:05,158] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:57:05,406] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:05,413] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:05,517] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:57:05,824] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:06,151] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:06,152] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:57:06,398] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:06,407] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:06,509] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:57:06,806] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:07,134] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:07,134] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:57:07,379] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:07,387] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:07,521] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:57:07,796] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:08,222] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:08,223] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 04:57:08,705] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:08,733] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:57:08,733] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:57:08,743] - [rest_client:2621] INFO - Node 172.23.105.126 not part of cluster inactiveFailed [2022-03-01 04:57:08,744] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:57:09,132] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:09,270] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:57:09,538] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:09,918] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:09,919] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:57:10,175] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:10,176] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:57:10,178] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:10,319] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:57:10,599] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:10,980] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:10,981] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:57:12,216] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:12,221] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:57:12,224] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:12,360] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:57:12,632] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:13,017] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:13,018] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:57:13,292] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:13,293] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:57:13,295] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:13,439] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:57:13,715] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:14,109] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:14,110] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:57:15,294] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:15,299] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:57:15,301] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:15,405] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:57:15,717] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:16,052] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:16,053] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:57:16,228] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:16,229] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:57:16,231] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:16,332] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:57:16,638] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:16,966] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:16,967] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:57:17,502] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:17,507] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:57:17,509] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:17,615] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:57:17,898] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:18,234] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:18,234] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:57:18,412] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:18,413] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:57:18,415] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:18,517] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:57:18,796] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:19,134] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:19,134] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:57:19,686] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:19,692] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:57:19,694] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:19,801] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:57:20,137] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:20,543] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:20,543] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:57:20,718] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:20,719] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:57:20,721] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:20,826] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:57:21,136] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:21,471] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:21,472] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:57:22,015] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:22,020] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:57:22,020] - [basetestcase:829] INFO - sleep for 5 secs. after failover before invoking rebalance... ... [2022-03-01 04:57:27,026] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.126', 'user': 'Administrator', 'password': 'password'} [2022-03-01 04:57:27,089] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 04:57:27,092] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 04:57:47,120] - [rest_client:1951] INFO - rebalance progress took 20.03 seconds [2022-03-01 04:57:47,121] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 04:57:57,125] - [failovertests:197] INFO - Begin VERIFICATION for Rebalance after Failover Only [2022-03-01 04:57:57,149] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:57:57,159] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:57:57,170] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:57:57,180] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:57:58,063] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:58,200] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:57:58,480] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:58,869] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:58,869] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:57:59,135] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:57:59,136] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:57:59,138] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:57:59,279] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:57:59,547] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:57:59,927] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:57:59,928] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:58:01,621] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:01,629] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.104.172:8091',default bucket [2022-03-01 04:58:01,631] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:01,769] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:58:02,045] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:02,436] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:02,437] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:02,752] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:02,753] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:58:02,755] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:02,890] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:58:03,173] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:03,567] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:03,568] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:58:05,250] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:05,257] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.106.52:8091',default bucket [2022-03-01 04:58:05,259] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:05,363] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:58:05,674] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:06,007] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:06,008] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:06,175] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:06,177] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:58:06,179] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:06,286] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:58:06,593] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:06,922] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:06,923] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:58:07,658] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:07,665] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.158:8091',default bucket [2022-03-01 04:58:07,667] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:07,773] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:58:08,065] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:08,416] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:08,417] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:08,587] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:08,589] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:58:08,591] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:08,709] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:58:08,994] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:09,330] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:09,331] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:58:10,084] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:10,091] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.127:8091',default bucket [2022-03-01 04:58:10,093] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:10,197] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:58:10,505] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:10,836] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:10,837] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:11,009] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:11,010] - [task:972] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:58:11,012] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:11,109] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:58:11,416] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:11,743] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:11,743] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default dcp [2022-03-01 04:58:12,464] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:12,471] - [task:972] INFO - Saw ep_dcp_items_remaining 0 == 0 expected on '172.23.105.162:8091',default bucket [2022-03-01 04:58:12,640] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 04:58:12,731] - [data_helper:308] INFO - creating direct client 172.23.105.127:11210 default [2022-03-01 04:58:12,800] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 04:58:12,963] - [data_helper:308] INFO - creating direct client 172.23.105.162:11210 default [2022-03-01 04:58:13,045] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 04:58:13,195] - [task:2319] INFO - 10000 items will be verified on default bucket [2022-03-01 04:58:13,474] - [task:2328] INFO - 0 items were verified [2022-03-01 04:58:16,839] - [task:2328] INFO - 10000 items were verified [2022-03-01 04:58:16,839] - [task:2332] INFO - 10000 items were verified in 3.644113302230835 sec.the average number of ops - 2744.0 per second [2022-03-01 04:58:16,865] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:58:16,876] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:58:17,501] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:17,637] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:58:17,907] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:18,289] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:18,290] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:18,581] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:18,584] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:18,721] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:58:19,009] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:19,399] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:19,400] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:19,702] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:19,705] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:19,809] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:58:20,125] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:20,456] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:20,457] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:20,633] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:20,637] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:20,741] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:58:21,029] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:21,361] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:21,362] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:21,532] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:21,535] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:21,643] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:58:21,947] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:22,277] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:22,278] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:22,450] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:22,451] - [task:972] INFO - Saw curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:58:22,454] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:22,587] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:58:22,861] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:23,241] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:23,243] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:23,537] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:23,540] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:23,680] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:58:23,950] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:24,341] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:24,342] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:24,671] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:24,675] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:24,780] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:58:25,096] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:25,426] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:25,427] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:25,600] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:25,603] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:25,705] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:58:25,986] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:26,322] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:26,323] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:26,491] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:26,494] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:26,592] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:58:26,901] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:27,233] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:27,234] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:27,401] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:27,402] - [task:972] INFO - Saw vb_active_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:58:27,404] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:27,537] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:58:27,804] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:28,184] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:28,185] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:28,480] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:28,483] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:28,616] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:58:28,889] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:29,273] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:29,275] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:29,570] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:29,573] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:29,677] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:58:29,991] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:30,326] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:30,327] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:30,501] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:30,504] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:30,608] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:58:30,893] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:31,241] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:31,242] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:31,422] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:31,425] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:31,529] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:58:31,843] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:32,182] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:32,183] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:32,359] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:32,360] - [task:972] INFO - Saw vb_replica_curr_items 10000 == 10000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:58:32,362] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:32,501] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:58:32,775] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:33,164] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:33,165] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:33,465] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:33,469] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:33,601] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:58:33,872] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:34,257] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:34,258] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:34,554] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:34,557] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:34,660] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:58:34,977] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:35,314] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:35,315] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:35,486] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:35,489] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:35,590] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:58:35,878] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:36,214] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:36,215] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:36,386] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:36,389] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:58:36,492] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:58:36,798] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:58:37,125] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:58:37,125] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default all [2022-03-01 04:58:37,294] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:58:37,295] - [task:972] INFO - Saw curr_items_tot 20000 == 20000 expected on '172.23.104.172:8091''172.23.106.52:8091''172.23.105.158:8091''172.23.105.127:8091''172.23.105.162:8091',default bucket [2022-03-01 04:58:37,296] - [rebalance_helper:272] INFO - waiting for sum_of_curr_items == total_items.... [2022-03-01 04:58:38,665] - [rebalance_helper:370] INFO - curr_items from 172.23.104.172:8091 : 2005 [2022-03-01 04:58:38,665] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.104.172:8091 : 0 [2022-03-01 04:58:38,666] - [rebalance_helper:380] INFO - vb_active_num from 172.23.104.172:8091 : 205 [2022-03-01 04:58:38,666] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.104.172:8091 : 205 [2022-03-01 04:58:38,666] - [rebalance_helper:370] INFO - curr_items from 172.23.105.127:8091 : 2006 [2022-03-01 04:58:38,666] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.127:8091 : 0 [2022-03-01 04:58:38,666] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.127:8091 : 205 [2022-03-01 04:58:38,666] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.127:8091 : 205 [2022-03-01 04:58:38,666] - [rebalance_helper:370] INFO - curr_items from 172.23.105.158:8091 : 1982 [2022-03-01 04:58:38,666] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.158:8091 : 0 [2022-03-01 04:58:38,666] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.158:8091 : 205 [2022-03-01 04:58:38,667] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.158:8091 : 205 [2022-03-01 04:58:38,667] - [rebalance_helper:370] INFO - curr_items from 172.23.105.162:8091 : 2026 [2022-03-01 04:58:38,667] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.105.162:8091 : 0 [2022-03-01 04:58:38,667] - [rebalance_helper:380] INFO - vb_active_num from 172.23.105.162:8091 : 205 [2022-03-01 04:58:38,667] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.105.162:8091 : 205 [2022-03-01 04:58:38,667] - [rebalance_helper:370] INFO - curr_items from 172.23.106.52:8091 : 1981 [2022-03-01 04:58:38,667] - [rebalance_helper:375] INFO - vb_pending_num from 172.23.106.52:8091 : 0 [2022-03-01 04:58:38,667] - [rebalance_helper:380] INFO - vb_active_num from 172.23.106.52:8091 : 204 [2022-03-01 04:58:38,667] - [rebalance_helper:385] INFO - vb_replica_num from 172.23.106.52:8091 : 204 [2022-03-01 04:58:38,668] - [rebalance_helper:388] INFO - summation of vb_active_num : 1024 vb_pending_num : 0 vb_replica_num : 1024 [2022-03-01 04:58:38,668] - [rebalance_helper:390] INFO - sum : 10000 and sum * (replica_factor + 1) (2) : 20000 [2022-03-01 04:58:39,064] - [rebalance_helper:393] INFO - curr_items_tot from master: 20000 [2022-03-01 04:58:39,065] - [rebalance_helper:412] INFO - delta : 0 missing_percentage : 1 replica_factor : 1 [2022-03-01 04:58:39,067] - [basetestcase:829] INFO - sleep for 60 secs. ... [2022-03-01 04:59:39,091] - [basetestcase:2135] INFO - Begin Verification for data comparison [2022-03-01 04:59:39,115] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 04:59:39,126] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 04:59:39,138] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:39,276] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:59:39,549] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:39,932] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:39,933] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:59:41,261] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:59:41,504] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:41,641] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:59:41,921] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:42,302] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:42,303] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:59:43,600] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:59:43,846] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:43,950] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:59:44,260] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:44,590] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:44,591] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:59:45,160] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:59:45,379] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:45,480] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:59:45,771] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:46,106] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:46,107] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:59:46,674] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:59:46,887] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:46,990] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:59:47,303] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:47,638] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:47,639] - [remote_util:4356] INFO - Run cbtransfer to get data map [2022-03-01 04:59:48,204] - [remote_util:4404] INFO - done execute cbtransfer [2022-03-01 04:59:48,477] - [basetestcase:2144] INFO - End Verification for data comparison [2022-03-01 04:59:48,479] - [basetestcase:2301] INFO - Begin Verification for failovers logs comparison [2022-03-01 04:59:48,483] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:48,619] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:59:48,900] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:49,278] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:49,279] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:49,558] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:49,559] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:49,798] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:49,801] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:49,944] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:59:50,215] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:50,597] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:50,598] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:50,849] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:50,850] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:51,129] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:51,132] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:51,238] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:59:51,557] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:51,892] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:51,893] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:52,025] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:52,026] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:52,192] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:52,196] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:52,297] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:59:52,581] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:52,924] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:52,925] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:53,098] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:53,099] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:53,230] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:53,233] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:53,336] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:59:53,643] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:53,979] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:53,980] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:54,148] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:54,149] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 04:59:54,273] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:54,276] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:54,410] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 04:59:54,682] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:55,064] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:55,065] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:59:55,423] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:55,430] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:55,564] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 04:59:55,834] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:56,239] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:56,240] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:59:56,585] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:56,593] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:56,795] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 04:59:57,321] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:57,963] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:57,964] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:59:58,250] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:58,258] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:58,557] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 04:59:58,843] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 04:59:59,182] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 04:59:59,183] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 04:59:59,385] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 04:59:59,393] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 04:59:59,495] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 04:59:59,807] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:00,142] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:00,143] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default failovers [2022-03-01 05:00:00,347] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:00,418] - [basetestcase:2308] INFO - End Verification for Failovers logs comparison [2022-03-01 05:00:00,418] - [basetestcase:1863] INFO - Begin Verification for vbucket sequence numbers comparison [2022-03-01 05:00:00,420] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:00,556] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:00:00,848] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:01,242] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:01,243] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:01,756] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:01,766] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:01,922] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:00:02,217] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:02,626] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:02,627] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:03,184] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:03,194] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:03,301] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:00:03,610] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:03,946] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:03,947] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:04,209] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:04,220] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:04,324] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:00:04,609] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:04,952] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:04,952] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:05,219] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:05,229] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:05,337] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:00:05,650] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:05,988] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:05,988] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:06,256] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:06,326] - [basetestcase:1888] INFO - End Verification for vbucket sequence numbers comparison [2022-03-01 05:00:06,382] - [basetestcase:2056] INFO - Begin Verification for vb_distribution_analysis [2022-03-01 05:00:06,394] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 05:00:06,405] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 05:00:06,415] - [rest_client:2647] INFO - Node versions in cluster ['7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise', '7.1.0-2407-enterprise'] [2022-03-01 05:00:06,425] - [basetestcase:2641] INFO - list of kv nodes in cluster: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 05:00:06,427] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:06,572] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:00:06,862] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:07,262] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:07,263] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:07,518] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:07,519] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:07,809] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:07,812] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:07,916] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:00:08,194] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:08,521] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:08,522] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:08,644] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:08,645] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:08,770] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:08,773] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:08,877] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:00:09,186] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:09,518] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:09,519] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:09,686] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:09,686] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:09,851] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:09,854] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:09,962] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:00:10,267] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:10,597] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:10,598] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:10,759] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:10,760] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:10,883] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:10,886] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:11,024] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:00:11,312] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:11,712] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:11,713] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:11,971] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:11,971] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket [2022-03-01 05:00:12,221] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:12,222] - [basetestcase:2064] INFO - Begin Verification for Bucket default [2022-03-01 05:00:12,281] - [basetestcase:2094] INFO - End Verification for vb_distribution_analysis [2022-03-01 05:00:12,281] - [failovertests:219] INFO - End VERIFICATION for Rebalance after Failover Only [2022-03-01 05:00:12,293] - [basetestcase:3028] INFO - Servers to check bucket-seqno: [ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot, ip:172.23.105.162 port:8091 ssh_username:nonroot, ip:172.23.106.52 port:8091 ssh_username:nonroot] [2022-03-01 05:00:12,293] - [basetestcase:2985] INFO - Record vbucket for the bucket default [2022-03-01 05:00:12,432] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:12,571] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:00:12,849] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:13,257] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:13,258] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:13,751] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:13,760] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:13,860] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:00:14,151] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:14,490] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:14,491] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:14,767] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:14,776] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:14,879] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:00:15,186] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:15,517] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:15,518] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:15,778] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:15,786] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:15,890] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:00:16,188] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:16,515] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:16,516] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:16,769] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:16,778] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:16,915] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:00:17,193] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:17,585] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:17,586] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: ~/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p 'password' -b default vbucket-seqno [2022-03-01 05:00:18,118] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:18,134] - [failoverbasetests:98] INFO - ============== tearDown was started for test #12 test_failover_normal ============== [2022-03-01 05:00:18,136] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:18,272] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:00:18,543] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:18,929] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:18,930] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:00:19,058] - [remote_util:177] INFO - process beam.smp is running on 172.23.104.172: with pid 130199 [2022-03-01 05:00:19,058] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2022-03-01 05:00:19,198] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2022-03-01 05:00:19,199] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:00:19,199] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -F [2022-03-01 05:00:19,388] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:19,388] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -t nat -F [2022-03-01 05:00:19,450] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:19,450] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:19,585] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:00:19,585] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 05:00:19,585] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT memcached [2022-03-01 05:00:19,793] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 05:00:19,793] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 05:00:19,793] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: killall -SIGCONT beam [2022-03-01 05:00:19,877] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 05:00:19,877] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 05:00:19,880] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:20,030] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:00:20,339] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:20,811] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:20,811] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:00:20,959] - [remote_util:177] INFO - process beam.smp is running on 172.23.106.52: with pid 27929 [2022-03-01 05:00:20,960] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:root, attempt#1 of 5 [2022-03-01 05:00:21,155] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as root [2022-03-01 05:00:21,156] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:00:21,156] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -F [2022-03-01 05:00:21,455] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:21,455] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: /sbin/iptables -t nat -F [2022-03-01 05:00:21,585] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:21,585] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:21,813] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:00:21,813] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 05:00:21,813] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT memcached [2022-03-01 05:00:22,058] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 05:00:22,058] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 05:00:22,058] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: killall -SIGCONT beam [2022-03-01 05:00:22,138] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 05:00:22,138] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 05:00:22,141] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:22,243] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:00:22,549] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:22,880] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:22,880] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:00:22,967] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.158: with pid 25310 [2022-03-01 05:00:22,967] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:root, attempt#1 of 5 [2022-03-01 05:00:23,070] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as root [2022-03-01 05:00:23,070] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:00:23,070] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -F [2022-03-01 05:00:23,290] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:23,290] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: /sbin/iptables -t nat -F [2022-03-01 05:00:23,346] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:23,346] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:23,448] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:00:23,448] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 05:00:23,449] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT memcached [2022-03-01 05:00:23,685] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 05:00:23,685] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 05:00:23,685] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: killall -SIGCONT beam [2022-03-01 05:00:23,747] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 05:00:23,747] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 05:00:23,749] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:23,855] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:00:24,169] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:24,510] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:24,511] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:00:24,597] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.126: with pid 10785 [2022-03-01 05:00:24,597] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:root, attempt#1 of 5 [2022-03-01 05:00:24,700] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as root [2022-03-01 05:00:24,701] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:00:24,701] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -F [2022-03-01 05:00:24,924] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:24,925] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: /sbin/iptables -t nat -F [2022-03-01 05:00:24,982] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:24,983] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:25,085] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:00:25,085] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 05:00:25,085] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT memcached [2022-03-01 05:00:25,330] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 05:00:25,330] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 05:00:25,330] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: killall -SIGCONT beam [2022-03-01 05:00:25,397] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 05:00:25,397] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 05:00:25,399] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:25,506] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:00:25,794] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:26,133] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:26,134] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:00:26,219] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.127: with pid 42970 [2022-03-01 05:00:26,220] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:root, attempt#1 of 5 [2022-03-01 05:00:26,325] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as root [2022-03-01 05:00:26,325] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:00:26,325] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -F [2022-03-01 05:00:26,527] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:26,527] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: /sbin/iptables -t nat -F [2022-03-01 05:00:26,583] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:26,583] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:26,687] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:00:26,687] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 05:00:26,687] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT memcached [2022-03-01 05:00:26,910] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 05:00:26,910] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 05:00:26,911] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: killall -SIGCONT beam [2022-03-01 05:00:26,977] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 05:00:26,978] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 05:00:26,980] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:27,084] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:00:27,388] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:00:27,722] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:00:27,723] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:00:27,812] - [remote_util:177] INFO - process beam.smp is running on 172.23.105.162: with pid 21904 [2022-03-01 05:00:27,812] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:root, attempt#1 of 5 [2022-03-01 05:00:27,914] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as root [2022-03-01 05:00:27,915] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:00:27,915] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -F [2022-03-01 05:00:28,139] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:28,139] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: /sbin/iptables -t nat -F [2022-03-01 05:00:28,194] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:00:28,195] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:00:28,293] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:00:28,294] - [remote_util:3992] INFO - *** unpause memcached process *** [2022-03-01 05:00:28,294] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT memcached [2022-03-01 05:00:28,532] - [remote_util:3381] INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2022-03-01 05:00:28,532] - [remote_util:3114] ERROR - memcached: no process found [2022-03-01 05:00:28,532] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: killall -SIGCONT beam [2022-03-01 05:00:28,596] - [remote_util:3381] INFO - command executed with nonroot but got an error ['beam: no process found'] ... [2022-03-01 05:00:28,596] - [remote_util:3114] ERROR - beam: no process found [2022-03-01 05:00:38,730] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 05:00:39,906] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 05:00:39,911] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 05:00:39,911] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 05:00:39,922] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 05:00:39,929] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 05:00:39,936] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 05:00:39,943] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 05:00:39,950] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 05:00:39,959] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:00:39,988] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 05:00:39,991] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.127,ns_1@172.23.105.158,ns_1@172.23.105.162,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 05:00:40,046] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 05:00:40,049] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 05:00:50,056] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 05:01:00,070] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 05:01:20,099] - [rest_client:1951] INFO - rebalance progress took 40.05 seconds [2022-03-01 05:01:20,099] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:01:30,134] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.127', 8091), ('ns_1@172.23.105.158', 8091), ('ns_1@172.23.105.162', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 05:01:30,138] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:30,144] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:30,155] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:30,160] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:30,169] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:30,173] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:30,181] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:30,185] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:30,193] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:30,198] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:30,203] - [failoverbasetests:109] INFO - ============== tearDown was finished for test #12 test_failover_normal ============== [2022-03-01 05:01:30,216] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 05:01:30,216] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.28625472887768, 'mem_free': 2969538560, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 05:01:30,216] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 05:01:30,218] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:30,355] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:01:30,628] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:31,008] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:31,011] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:31,147] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:01:31,424] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:31,814] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:31,817] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:31,923] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:01:32,243] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:32,574] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:32,577] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:32,680] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:01:32,991] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:33,324] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:33,326] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:33,431] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:01:33,711] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:34,046] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:34,049] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:34,157] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:01:34,471] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:34,803] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:48,065] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #12 test_failover_normal ============== [2022-03-01 05:01:48,090] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 05:01:48,099] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 05:01:48,104] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 05:01:48,110] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 05:01:48,115] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 05:01:48,120] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 05:01:48,120] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 05:01:48,132] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 05:01:48,133] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 05:01:48,139] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,162] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 05:01:48,162] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,167] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 05:01:48,172] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 05:01:48,172] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,176] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 05:01:48,180] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 05:01:48,180] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,184] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 05:01:48,188] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 05:01:48,188] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,190] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 05:01:48,193] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 05:01:48,194] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,196] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 05:01:48,199] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 05:01:48,199] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:48,202] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 05:01:48,202] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #12 test_failover_normal ============== [2022-03-01 05:01:48,202] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 05:01:48,202] - [basetestcase:760] INFO - closing all memcached connections Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 summary so far suite failover.failovertests.FailoverTests , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_12 ok ---------------------------------------------------------------------- Ran 1 test in 1102.095s OK test_autofailover (failover.AutoFailoverTests.AutoFailoverTests) ... Logs will be stored at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_13 ./testrunner -i /tmp/testexec.30028.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False -t failover.AutoFailoverTests.AutoFailoverTests.test_autofailover,timeout=5,num_node_failures=1,failover_orchestrator=True,failover_action=stop_server,nodes_init=3 Test Input params: {'timeout': '5', 'num_node_failures': '1', 'failover_orchestrator': 'True', 'failover_action': 'stop_server', 'nodes_init': '3', 'ini': '/tmp/testexec.30028.ini', 'cluster_name': 'testexec.30028', 'spec': 'py-rebalance-extended-sanity', 'conf_file': 'conf/py-rebalance-extended-sanity.conf', 'get-cbcollect-info': 'False', 'get-logs': 'False', 'get-coredumps': 'False', 'num_nodes': 6, 'case_number': 13, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_13'} [2022-03-01 05:01:48,234] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:48,370] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:01:48,650] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:49,027] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:49,035] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:01:49,053] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 05:01:49,053] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #13 test_autofailover============== [2022-03-01 05:01:49,053] - [basetestcase:331] INFO - initializing cluster [2022-03-01 05:01:49,240] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:01:49,246] - [task:159] INFO - {'uptime': '1917', 'memoryTotal': 3947372544, 'memoryFree': 2975875072, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 05:01:49,253] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 05:01:49,253] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 05:01:49,253] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 05:01:49,264] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 05:01:49,264] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 05:01:49,264] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 05:01:49,279] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 05:01:49,294] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 05:01:49,294] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 05:01:49,297] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091//node/controller/setupServices body: hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:password [2022-03-01 05:01:49,297] - [rest_client:1231] INFO - This node is already provisioned with services, we do not consider this as failure for test case [2022-03-01 05:01:49,297] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:01:49,297] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 05:01:49,430] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:01:49,434] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:49,583] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:01:49,854] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:50,236] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:50,237] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:01:50,279] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:01:50,280] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:01:50,283] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:50,287] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:50,296] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:01:50,329] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:01:50,333] - [task:159] INFO - {'uptime': '6664', 'memoryTotal': 3947372544, 'memoryFree': 3217866752, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 05:01:50,336] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:50,336] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 05:01:50,342] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:01:50,342] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 05:01:50,477] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:01:50,479] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:50,619] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:01:50,893] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:51,296] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:51,297] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:01:51,380] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:01:51,380] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:01:51,384] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:51,388] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:51,394] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:01:51,417] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:01:51,419] - [task:159] INFO - {'uptime': '6640', 'memoryTotal': 4201627648, 'memoryFree': 3534524416, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:01:51,421] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:51,421] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:01:51,425] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:01:51,425] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 05:01:51,482] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:01:51,484] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:51,589] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:01:51,901] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:52,235] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:52,236] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:01:52,307] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:01:52,307] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:01:52,310] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:52,312] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:52,316] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:01:52,330] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:01:52,333] - [task:159] INFO - {'uptime': '6642', 'memoryTotal': 4201627648, 'memoryFree': 3520077824, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:01:52,335] - [rest_client:1073] ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:52,335] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:01:52,338] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:01:52,339] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 05:01:52,442] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:01:52,444] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:52,550] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:01:52,860] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:53,205] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:53,206] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:01:53,275] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:01:53,275] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:01:53,278] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:53,280] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:53,283] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:01:53,297] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:01:53,300] - [task:159] INFO - {'uptime': '6641', 'memoryTotal': 4201627648, 'memoryFree': 3506339840, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:01:53,302] - [rest_client:1073] ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:53,302] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:01:53,305] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:01:53,306] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 05:01:53,364] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:01:53,365] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:53,470] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:01:53,767] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:54,103] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:54,104] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:01:54,172] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:01:54,172] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:01:54,175] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:54,178] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:54,181] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:01:54,195] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:01:54,198] - [task:159] INFO - {'uptime': '6640', 'memoryTotal': 4201627648, 'memoryFree': 3507486720, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:01:54,200] - [rest_client:1073] ERROR - GET http://172.23.105.162:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:01:54,200] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:01:54,203] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:01:54,204] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 05:01:54,264] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:01:54,266] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:01:54,369] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:01:54,665] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:01:54,993] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:01:54,995] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:01:55,063] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:01:55,063] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:01:55,066] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:55,068] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:01:55,074] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:01:55,084] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 05:01:55,252] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 05:02:00,255] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 05:02:00,274] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 05:02:01,091] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 05:02:01,101] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 05:02:11,118] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 05:02:11,118] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:02:27,518] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 05:02:27,531] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 05:02:37,547] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 05:02:37,547] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:02:51,216] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 05:02:51,217] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 05:02:51,227] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 05:02:51,280] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 05:03:01,294] - [rest_client:2012] INFO - rebalance percentage : 27.00 % [2022-03-01 05:03:01,294] - [task:843] INFO - Rebalance - status: running, progress: 27.00% [2022-03-01 05:03:21,330] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 05:03:21,339] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.05915927886963 sec [2022-03-01 05:03:21,341] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:21,476] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:03:21,751] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:22,127] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:22,128] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:03:22,207] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:03:22,208] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:03:22,208] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 05:03:22,366] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 05:03:22,399] - [rest_client:3014] INFO - 0.03 seconds to create bucket default [2022-03-01 05:03:22,399] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 05:03:23,145] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 05:03:23,386] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 05:03:23,514] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 05:03:23,823] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 05:03:24,040] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 05:03:24,183] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 05:03:25,068] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 05:03:25,070] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #13 test_autofailover ============== [2022-03-01 05:03:25,081] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:25,229] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:03:25,514] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:25,934] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:25,937] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:26,107] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:03:26,391] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:26,789] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:26,792] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:26,899] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:03:27,214] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:27,547] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:27,549] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:27,654] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:03:27,956] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:28,294] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:28,298] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:28,401] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:03:28,682] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:29,013] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:29,016] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:29,119] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:03:29,424] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:29,752] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:41,927] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 05:03:41,928] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 23.71828068358364, 'mem_free': 2910707712, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 05:03:41,928] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 3.430012610340479, 'mem_free': 3354226688, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 05:03:41,928] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 9.85021579080985, 'mem_free': 3037999104, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 05:03:41,928] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 05:03:41,928] - [basetestcase:559] INFO - Time to execute basesetup : 113.69601559638977 [2022-03-01 05:03:41,936] - [rest_client:3099] INFO - settings/autoFailover params : timeout=5&enabled=true&canAbortRebalance=true&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2022-03-01 05:03:41,944] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 05:03:46,953] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 05:03:51,958] - [task:6365] INFO - Initializing NodeDownTimerTask [2022-03-01 05:03:52,959] - [task:6082] INFO - before failure time: Tue Mar 1 05:03:52 2022 [2022-03-01 05:03:53,962] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:03:54,100] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:03:54,373] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:03:54,763] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:03:54,764] - [remote_util:3902] INFO - Stop Couchbase Server with non root method [2022-03-01 05:03:54,764] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /home/nonroot//opt/couchbase/bin/couchbase-server -k [2022-03-01 05:03:54,951] - [task:6372] INFO - Starting execution of NodeDownTimerTask [2022-03-01 05:03:56,792] - [task:6413] INFO - Injected failure in 172.23.104.172 [2022-03-01 05:03:56,842] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:03:56,842] - [task:6154] INFO - Stopped the couchbase server on ip:172.23.104.172 port:8091 ssh_username:nonroot [2022-03-01 05:03:56,842] - [task:6123] INFO - Start time = Tue Mar 1 05:03:55 2022 [2022-03-01 05:04:13,285] - [task:6029] INFO - Autofailover of node 172.23.104.172 successfully initiated in 17.208977222442627 sec [2022-03-01 05:04:13,305] - [rest_client:3969] INFO - Latest logs from UI on 172.23.106.52: [2022-03-01 05:04:13,305] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.105.158', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1646139853258, 'shortText': 'message', 'text': "Starting failover of nodes ['ns_1@172.23.104.172']. Operation Id = 5c8255f5b551d3898cae99c3aafe31b9", 'serverTime': '2022-03-01T05:04:13.258Z'} [2022-03-01 05:04:13,305] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.105.158', 'type': 'info', 'code': 0, 'module': 'failover', 'tstamp': 1646139853258, 'shortText': 'message', 'text': "Starting failing over ['ns_1@172.23.104.172']", 'serverTime': '2022-03-01T05:04:13.258Z'} [2022-03-01 05:04:13,305] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.105.158', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1646139847236, 'shortText': 'message', 'text': 'Enabled auto-failover with timeout 5 and max count 1', 'serverTime': '2022-03-01T05:04:07.236Z'} [2022-03-01 05:04:13,305] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.105.158', 'type': 'info', 'code': 0, 'module': 'mb_master', 'tstamp': 1646139847231, 'shortText': 'message', 'text': "Haven't heard from a higher priority node or a master, so I'm taking over.", 'serverTime': '2022-03-01T05:04:07.231Z'} [2022-03-01 05:04:13,306] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.105.158', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1646139836979, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.105.158' saw that node 'ns_1@172.23.104.172' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-01T05:03:56.979Z'} [2022-03-01 05:04:13,306] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.106.52', 'type': 'warning', 'code': 5, 'module': 'ns_node_disco', 'tstamp': 1646139836977, 'shortText': 'node down', 'text': "Node 'ns_1@172.23.106.52' saw that node 'ns_1@172.23.104.172' went down. Details: [{nodedown_reason,\n connection_closed}]", 'serverTime': '2022-03-01T05:03:56.977Z'} [2022-03-01 05:04:13,306] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.104.172', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1646139836802, 'shortText': 'message', 'text': "Control connection to memcached on 'ns_1@172.23.104.172' disconnected. Check logs for details.", 'serverTime': '2022-03-01T05:03:56.802Z'} [2022-03-01 05:04:13,306] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.104.172', 'type': 'info', 'code': 0, 'module': 'auto_failover', 'tstamp': 1646139821940, 'shortText': 'message', 'text': 'Updating auto-failover timeout to 5', 'serverTime': '2022-03-01T05:03:41.940Z'} [2022-03-01 05:04:13,306] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.104.172', 'type': 'warning', 'code': 102, 'module': 'menelaus_web', 'tstamp': 1646139805079, 'shortText': 'client-side error report', 'text': 'Client-side error-report for user "Administrator" on node \'ns_1@172.23.104.172\':\nUser-Agent:Python-httplib2/0.13.1 (gzip)\n2022-03-01 05:03:25.070667 : test_autofailover started ', 'serverTime': '2022-03-01T05:03:25.079Z'} [2022-03-01 05:04:13,306] - [rest_client:3970] ERROR - {'node': 'ns_1@172.23.104.172', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1646139802979, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.104.172\' in 0 seconds.', 'serverTime': '2022-03-01T05:03:22.979Z'} [2022-03-01 05:04:13,307] - [rest_client:3099] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2022-03-01 05:04:14,127] - [AutoFailoverBaseTest:89] INFO - ============AutoFailoverBaseTest teardown============ [2022-03-01 05:04:15,140] - [task:6082] INFO - before failure time: Tue Mar 1 05:04:15 2022 [2022-03-01 05:04:15,142] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:04:15,285] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:04:15,584] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:04:15,964] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:04:15,965] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:04:16,082] - [remote_util:3931] INFO - Starting couchbase server [2022-03-01 05:04:16,083] - [remote_util:3941] INFO - Start Couchbase Server with non root method [2022-03-01 05:04:16,083] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /home/nonroot//opt/couchbase/bin/couchbase-server \-- -noinput -detached [2022-03-01 05:04:17,485] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:04:17,485] - [remote_util:3169] INFO - [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - The maximum number of processes for the couchbase user is set too low (4096). [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - It must be at least 10000. [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - Normally this can be increased by adding the following lines to [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - /etc/security/limits.conf: [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - couchbase soft nproc [2022-03-01 05:04:17,486] - [remote_util:3169] INFO - couchbase hard nproc [2022-03-01 05:04:17,487] - [remote_util:3169] INFO - [2022-03-01 05:04:17,487] - [remote_util:3169] INFO - Where is greater than 10000. The procedure may be totally [2022-03-01 05:04:17,487] - [remote_util:3169] INFO - different if you're running so called "non-root/non-sudo install" or [2022-03-01 05:04:17,487] - [remote_util:3169] INFO - if you've built Couchbase Server from source. [2022-03-01 05:04:17,487] - [remote_util:3169] INFO - [2022-03-01 05:04:17,487] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:04:17,586] - [remote_util:177] INFO - process beam.smp is running on 172.23.104.172: with pid 23014 [2022-03-01 05:04:17,587] - [task:6162] INFO - Started the couchbase server on ip:172.23.104.172 port:8091 ssh_username:nonroot [2022-03-01 05:04:17,587] - [task:6123] INFO - Start time = Wed Dec 31 16:00:00 1969 [2022-03-01 05:04:17,587] - [basetestcase:829] INFO - sleep for 10 secs. ... [2022-03-01 05:04:27,609] - [task:6082] INFO - before failure time: Tue Mar 1 05:04:27 2022 [2022-03-01 05:04:27,611] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:04:27,754] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:04:28,031] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:04:28,420] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:04:28,421] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2022-03-01 05:04:28,557] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2022-03-01 05:04:28,557] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:04:28,557] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -F [2022-03-01 05:04:28,737] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:04:28,737] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -t nat -F [2022-03-01 05:04:28,798] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:04:28,798] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:04:28,933] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:04:28,933] - [task:6123] INFO - Start time = Wed Dec 31 16:00:00 1969 [2022-03-01 05:04:28,946] - [rest_client:3099] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2022-03-01 05:04:28,951] - [rest_client:1078] ERROR - socket error while connecting to http://172.23.104.172:8091/nodes/self error [Errno 111] Connection refused [2022-03-01 05:04:32,006] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 05:04:32,836] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 05:04:32,840] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 05:04:32,840] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 05:04:32,851] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 05:04:32,858] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 05:04:32,866] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 05:04:32,875] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 05:04:32,887] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 05:04:32,894] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:04:32,907] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 05:04:32,929] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 05:04:32,932] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 05:04:32,941] - [rest_client:1073] ERROR - POST http://172.23.104.172:8091/controller/rebalance body: knownNodes=ns_1%40172.23.104.172%2Cns_1%40172.23.105.158%2Cns_1%40172.23.106.52&ejectedNodes=ns_1%40172.23.105.158%2Cns_1%40172.23.106.52&user=Administrator&password=password headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: status: 400, content: b'No active nodes left' b'No active nodes left' auth: Administrator:password [2022-03-01 05:04:32,942] - [rest_client:1852] ERROR - rebalance operation failed: b'No active nodes left' [2022-03-01 05:04:32,948] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:04:32,961] - [rest_client:2621] INFO - Node 172.23.104.172 not part of cluster inactiveFailed [2022-03-01 05:04:32,973] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 05:04:32,975] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158', 'user': 'Administrator', 'password': 'password'} [2022-03-01 05:04:33,031] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 05:04:33,035] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 05:04:43,051] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 05:04:53,067] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 05:05:13,096] - [rest_client:1951] INFO - rebalance progress took 40.07 seconds [2022-03-01 05:05:13,096] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:05:23,123] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.104.172', 8091), ('ns_1@172.23.105.158', 8091)] [2022-03-01 05:05:23,125] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:23,130] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:23,138] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:23,153] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:23,168] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:23,185] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:23,186] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 05:05:23,186] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 05:05:23,188] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:23,322] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:05:23,590] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:23,969] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:23,972] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:24,110] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:05:24,386] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:24,772] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:24,775] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:24,882] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:05:25,202] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:25,540] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:25,543] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:25,649] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:05:25,963] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:26,296] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:26,299] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:26,404] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:05:26,691] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:27,027] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:27,030] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:27,133] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:05:27,442] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:27,776] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:40,198] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #13 test_autofailover ============== [2022-03-01 05:05:40,206] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:40,214] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 05:05:40,225] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 05:05:40,231] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 05:05:40,236] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 05:05:40,241] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 05:05:40,246] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 05:05:40,246] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 05:05:40,252] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:40,257] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,263] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:40,273] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 05:05:40,274] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,278] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 05:05:40,284] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 05:05:40,284] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,289] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 05:05:40,292] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 05:05:40,293] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,296] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 05:05:40,300] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 05:05:40,300] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,303] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 05:05:40,306] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 05:05:40,306] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,309] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 05:05:40,312] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 05:05:40,312] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:40,315] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 05:05:40,315] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #13 test_autofailover ============== [2022-03-01 05:05:40,315] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 05:05:40,315] - [basetestcase:760] INFO - closing all memcached connections ok ---------------------------------------------------------------------- Ran 1 test in 232.090s OK suite_tearDown (failover.AutoFailoverTests.AutoFailoverTests) ... Cluster instance shutdown with force summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 summary so far suite failover.failovertests.FailoverTests , pass 4 , fail 0 summary so far suite failover.AutoFailoverTests.AutoFailoverTests , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_03-12-31/test_13 *** Tests executed count: 13 Run after suite setup for failover.AutoFailoverTests.AutoFailoverTests.test_autofailover [2022-03-01 05:05:40,327] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:40,462] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:05:40,735] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:41,111] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:41,117] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:05:41,129] - [rest_client:2637] INFO - Node version in cluster 7.1.0-2407-enterprise [2022-03-01 05:05:41,129] - [basetestcase:252] INFO - ============== basetestcase setup was started for test #13 suite_tearDown============== [2022-03-01 05:05:41,129] - [basetestcase:331] INFO - initializing cluster [2022-03-01 05:05:41,331] - [task:154] INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:05:41,335] - [task:159] INFO - {'uptime': '81', 'memoryTotal': 3947372544, 'memoryFree': 3272802304, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 05:05:41,337] - [rest_client:1073] ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:41,337] - [task:198] INFO - quota for index service will be 256 MB [2022-03-01 05:05:41,338] - [task:200] INFO - set index quota to node 172.23.104.172 [2022-03-01 05:05:41,338] - [rest_client:1260] INFO - pools/default params : indexMemoryQuota=256 [2022-03-01 05:05:41,343] - [task:203] INFO - quota for fts service will be 512 MB [2022-03-01 05:05:41,343] - [task:205] INFO - set both index and fts quota at node 172.23.104.172 [2022-03-01 05:05:41,343] - [rest_client:1260] INFO - pools/default params : ftsMemoryQuota=512 [2022-03-01 05:05:41,348] - [rest_client:1248] INFO - pools/default params : memoryQuota=1249 [2022-03-01 05:05:41,352] - [rest_client:1209] INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql', 'fts']) [2022-03-01 05:05:41,352] - [rest_client:1225] INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql%2Cfts [2022-03-01 05:05:41,380] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:05:41,381] - [rest_client:1149] INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2022-03-01 05:05:41,624] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:05:41,626] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:41,767] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:05:42,046] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:42,504] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:42,505] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:05:42,549] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:05:42,549] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:05:42,553] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:42,563] - [rest_client:1868] INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:42,574] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:05:42,605] - [task:154] INFO - server: ip:172.23.106.52 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:05:42,610] - [task:159] INFO - {'uptime': '6893', 'memoryTotal': 3947372544, 'memoryFree': 3050795008, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.106.52:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1249, 'memcached': 11210, 'id': 'ns_1@172.23.106.52', 'ip': '172.23.106.52', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['fts', 'index', 'kv', 'n1ql'], 'storageTotalRam': 3764, 'curr_items': 0} [2022-03-01 05:05:42,618] - [rest_client:1248] INFO - pools/default params : memoryQuota=2017 [2022-03-01 05:05:42,631] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:05:42,631] - [rest_client:1149] INFO - settings/web params on 172.23.106.52:8091:port=8091&username=Administrator&password=password [2022-03-01 05:05:42,757] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:05:42,759] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:42,898] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:05:43,176] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:43,556] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:43,557] - [remote_util:3334] INFO - running command.raw on 172.23.106.52: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:05:43,640] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:05:43,640] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:05:43,644] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:43,647] - [rest_client:1868] INFO - /diag/eval status on 172.23.106.52:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:43,656] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:05:43,692] - [task:154] INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:05:43,695] - [task:159] INFO - {'uptime': '6873', 'memoryTotal': 4201627648, 'memoryFree': 3529093120, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:05:43,697] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:05:43,697] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:05:43,701] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:05:43,702] - [rest_client:1149] INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2022-03-01 05:05:43,761] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:05:43,763] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:43,864] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:05:44,174] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:44,505] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:44,506] - [remote_util:3334] INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:05:44,575] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:05:44,575] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:05:44,578] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:44,580] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:44,584] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:05:44,597] - [task:154] INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:05:44,600] - [task:159] INFO - {'uptime': '6873', 'memoryTotal': 4201627648, 'memoryFree': 3508408320, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:05:44,605] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:05:44,608] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:05:44,608] - [rest_client:1149] INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2022-03-01 05:05:44,661] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:05:44,662] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:44,765] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:05:45,077] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:45,413] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:45,414] - [remote_util:3334] INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:05:45,483] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:05:45,483] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:05:45,486] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:45,489] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:45,493] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:05:45,507] - [task:154] INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:05:45,510] - [task:159] INFO - {'uptime': '6872', 'memoryTotal': 4201627648, 'memoryFree': 3487457280, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:05:45,513] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:05:45,517] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:05:45,517] - [rest_client:1149] INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2022-03-01 05:05:45,571] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:05:45,573] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:45,675] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:05:45,955] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:46,290] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:46,290] - [remote_util:3334] INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:05:46,359] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:05:46,360] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:05:46,363] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:46,365] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:46,369] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:05:46,384] - [task:154] INFO - server: ip:172.23.105.162 port:8091 ssh_username:nonroot, nodes/self [2022-03-01 05:05:46,387] - [task:159] INFO - {'uptime': '6872', 'memoryTotal': 4201627648, 'memoryFree': 3489267712, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.162:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-2407-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 2147, 'memcached': 11210, 'id': 'ns_1@172.23.105.162', 'ip': '172.23.105.162', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2022-03-01 05:05:46,391] - [rest_client:1248] INFO - pools/default params : memoryQuota=2147 [2022-03-01 05:05:46,394] - [rest_client:1144] INFO - --> in init_cluster...Administrator,password,8091 [2022-03-01 05:05:46,395] - [rest_client:1149] INFO - settings/web params on 172.23.105.162:8091:port=8091&username=Administrator&password=password [2022-03-01 05:05:46,447] - [rest_client:1151] INFO - --> status:True [2022-03-01 05:05:46,449] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:05:46,552] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:05:46,864] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:05:47,195] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:05:47,195] - [remote_util:3334] INFO - running command.raw on 172.23.105.162: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:05:47,266] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:05:47,266] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:05:47,269] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:47,271] - [rest_client:1868] INFO - /diag/eval status on 172.23.105.162:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2022-03-01 05:05:47,274] - [rest_client:1283] INFO - settings/indexes params : storageMode=plasma [2022-03-01 05:05:47,285] - [basetestcase:2378] INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2022-03-01 05:05:47,302] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 05:05:47,302] - [internal_user:36] INFO - Exception while deleting user. Exception is -b'"User was not found."' [2022-03-01 05:05:47,440] - [basetestcase:829] INFO - sleep for 5 secs. ... [2022-03-01 05:05:52,445] - [basetestcase:2383] INFO - **** add 'admin' role to 'cbadminbucket' user **** [2022-03-01 05:05:52,461] - [basetestcase:369] INFO - done initializing cluster [2022-03-01 05:05:53,292] - [task:773] INFO - adding node 172.23.106.52:8091 to cluster [2022-03-01 05:05:53,300] - [rest_client:1613] INFO - adding remote node @172.23.106.52:18091 to this cluster @172.23.104.172:8091 [2022-03-01 05:06:03,316] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 05:06:03,316] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:06:20,199] - [task:773] INFO - adding node 172.23.105.158:8091 to cluster [2022-03-01 05:06:20,211] - [rest_client:1613] INFO - adding remote node @172.23.105.158:18091 to this cluster @172.23.104.172:8091 [2022-03-01 05:06:30,227] - [rest_client:1951] INFO - rebalance progress took 10.02 seconds [2022-03-01 05:06:30,227] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:06:43,521] - [rest_client:2621] INFO - Node 172.23.105.158 not part of cluster inactiveAdded [2022-03-01 05:06:43,522] - [rest_client:2621] INFO - Node 172.23.106.52 not part of cluster inactiveAdded [2022-03-01 05:06:43,535] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2022-03-01 05:06:43,594] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 05:06:53,608] - [rest_client:2012] INFO - rebalance percentage : 27.00 % [2022-03-01 05:06:53,609] - [task:843] INFO - Rebalance - status: running, progress: 27.00% [2022-03-01 05:07:13,642] - [task:843] INFO - Rebalance - status: none, progress: 100.00% [2022-03-01 05:07:13,651] - [task:902] INFO - rebalancing was completed with progress: 100% in 30.056684732437134 sec [2022-03-01 05:07:13,653] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:13,793] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:07:14,066] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:14,445] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:14,446] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2022-03-01 05:07:14,524] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:07:14,524] - [remote_util:5219] INFO - ['ok'] [2022-03-01 05:07:14,525] - [basetestcase:2971] INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2022-03-01 05:07:14,675] - [rest_client:2989] INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1249&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2022-03-01 05:07:14,711] - [rest_client:3014] INFO - 0.04 seconds to create bucket default [2022-03-01 05:07:14,712] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2022-03-01 05:07:15,526] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 05:07:15,648] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 05:07:15,866] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 05:07:16,078] - [data_helper:308] INFO - creating direct client 172.23.104.172:11210 default [2022-03-01 05:07:16,205] - [data_helper:308] INFO - creating direct client 172.23.105.158:11210 default [2022-03-01 05:07:16,414] - [data_helper:308] INFO - creating direct client 172.23.106.52:11210 default [2022-03-01 05:07:17,173] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 1249 [2022-03-01 05:07:17,266] - [basetestcase:527] INFO - ============== basetestcase setup was finished for test #13 suite_tearDown ============== [2022-03-01 05:07:17,282] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:17,427] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:07:17,712] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:18,099] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:18,102] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:18,244] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:07:18,519] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:18,914] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:18,917] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:19,022] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:07:19,326] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:19,649] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:19,652] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:19,758] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:07:20,060] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:20,384] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:20,386] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:20,489] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:07:20,774] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:21,113] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:21,115] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:21,219] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:07:21,529] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:21,866] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:34,884] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 05:07:34,884] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 4.186633039092055, 'mem_free': 3073368064, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 05:07:34,885] - [basetestcase:585] INFO - 172.23.105.158:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 5.871931156669198, 'mem_free': 3392335872, 'mem_total': 4201627648, 'swap_mem_used': 54263808, 'swap_mem_total': 3758092288} [2022-03-01 05:07:34,885] - [basetestcase:585] INFO - 172.23.106.52:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 15.09627727856226, 'mem_free': 3070349312, 'mem_total': 3947372544, 'swap_mem_used': 67342336, 'swap_mem_total': 3758092288} [2022-03-01 05:07:34,885] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 05:07:34,885] - [basetestcase:559] INFO - Time to execute basesetup : 114.55971598625183 [2022-03-01 05:07:34,893] - [AutoFailoverBaseTest:89] INFO - ============AutoFailoverBaseTest teardown============ [2022-03-01 05:07:35,903] - [task:6082] INFO - before failure time: Tue Mar 1 05:07:35 2022 [2022-03-01 05:07:35,905] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:36,046] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:07:36,329] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:36,702] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:36,704] - [remote_util:158] INFO - Checking for process beam.smp on linux [2022-03-01 05:07:36,827] - [remote_util:177] INFO - process beam.smp is running on 172.23.104.172: with pid 23014 [2022-03-01 05:07:36,828] - [task:6162] INFO - Started the couchbase server on ip:172.23.104.172 port:8091 ssh_username:nonroot [2022-03-01 05:07:36,828] - [task:6123] INFO - Start time = Wed Dec 31 16:00:00 1969 [2022-03-01 05:07:36,828] - [basetestcase:829] INFO - sleep for 10 secs. ... [2022-03-01 05:07:46,850] - [task:6082] INFO - before failure time: Tue Mar 1 05:07:46 2022 [2022-03-01 05:07:46,851] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:46,989] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:07:47,267] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:07:47,652] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:07:47,653] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:root, attempt#1 of 5 [2022-03-01 05:07:47,789] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as root [2022-03-01 05:07:47,789] - [remote_util:1167] INFO - Non root or non sudo has no right to disable firewall, switching over to root [2022-03-01 05:07:47,789] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -F [2022-03-01 05:07:47,965] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:07:47,966] - [remote_util:3334] INFO - running command.raw on 172.23.104.172: /sbin/iptables -t nat -F [2022-03-01 05:07:48,026] - [remote_util:3383] INFO - command executed successfully with nonroot [2022-03-01 05:07:48,026] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:07:48,168] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:07:48,169] - [task:6123] INFO - Start time = Wed Dec 31 16:00:00 1969 [2022-03-01 05:07:48,181] - [rest_client:3099] INFO - settings/autoFailover params : timeout=120&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1 [2022-03-01 05:07:48,212] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 172.23.104.172 [2022-03-01 05:07:49,355] - [bucket_helper:218] INFO - waiting for bucket deletion to complete.... [2022-03-01 05:07:49,360] - [rest_client:140] INFO - node 172.23.104.172 existing buckets : [] [2022-03-01 05:07:49,360] - [bucket_helper:150] INFO - deleted bucket : default from 172.23.104.172 [2022-03-01 05:07:49,372] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 05:07:49,378] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 05:07:49,384] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 05:07:49,390] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 05:07:49,395] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 05:07:49,402] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:07:49,434] - [cluster_helper:270] INFO - rebalancing all nodes in order to remove nodes [2022-03-01 05:07:49,437] - [rest_client:1845] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.172,ns_1@172.23.105.158,ns_1@172.23.106.52', 'ejectedNodes': 'ns_1@172.23.105.158,ns_1@172.23.106.52', 'user': 'Administrator', 'password': 'password'} [2022-03-01 05:07:49,496] - [rest_client:1850] INFO - rebalance operation started [2022-03-01 05:07:49,499] - [rest_client:2012] INFO - rebalance percentage : 0.00 % [2022-03-01 05:07:59,513] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 05:08:09,528] - [rest_client:2012] INFO - rebalance percentage : 50.00 % [2022-03-01 05:08:29,557] - [rest_client:1951] INFO - rebalance progress took 40.06 seconds [2022-03-01 05:08:29,557] - [rest_client:1952] INFO - sleep for 10 seconds after rebalance... [2022-03-01 05:08:39,583] - [cluster_helper:333] INFO - removed all the nodes from cluster associated with ip:172.23.104.172 port:8091 ssh_username:nonroot ? [('ns_1@172.23.105.158', 8091), ('ns_1@172.23.106.52', 8091)] [2022-03-01 05:08:39,587] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:39,593] - [rest_client:1073] ERROR - GET http://172.23.106.52:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:08:39,602] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:39,606] - [rest_client:1073] ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2022-03-01 05:08:39,614] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:39,628] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:39,643] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:39,665] - [basetestcase:583] INFO - ------- Cluster statistics ------- [2022-03-01 05:08:39,665] - [basetestcase:585] INFO - 172.23.104.172:8091 => {'services': ['fts', 'index', 'kv', 'n1ql'], 'cpu_utilization': 1.332662811164194, 'mem_free': 3095261184, 'mem_total': 3947372544, 'swap_mem_used': 52850688, 'swap_mem_total': 3758092288} [2022-03-01 05:08:39,665] - [basetestcase:586] INFO - --- End of cluster statistics --- [2022-03-01 05:08:39,667] - [remote_util:302] INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2022-03-01 05:08:39,805] - [remote_util:338] INFO - SSH Connected to 172.23.104.172 as nonroot [2022-03-01 05:08:40,083] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:08:40,460] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:08:40,462] - [remote_util:302] INFO - SSH Connecting to 172.23.106.52 with username:nonroot, attempt#1 of 5 [2022-03-01 05:08:40,602] - [remote_util:338] INFO - SSH Connected to 172.23.106.52 as nonroot [2022-03-01 05:08:40,869] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:08:41,253] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:08:41,256] - [remote_util:302] INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2022-03-01 05:08:41,361] - [remote_util:338] INFO - SSH Connected to 172.23.105.158 as nonroot [2022-03-01 05:08:41,678] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:08:42,019] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:08:42,022] - [remote_util:302] INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2022-03-01 05:08:42,133] - [remote_util:338] INFO - SSH Connected to 172.23.105.126 as nonroot [2022-03-01 05:08:42,445] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:08:42,783] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:08:42,786] - [remote_util:302] INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2022-03-01 05:08:42,892] - [remote_util:338] INFO - SSH Connected to 172.23.105.127 as nonroot [2022-03-01 05:08:43,182] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:08:43,517] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:08:43,521] - [remote_util:302] INFO - SSH Connecting to 172.23.105.162 with username:nonroot, attempt#1 of 5 [2022-03-01 05:08:43,624] - [remote_util:338] INFO - SSH Connected to 172.23.105.162 as nonroot [2022-03-01 05:08:43,927] - [remote_util:3497] INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2022-03-01 05:08:44,256] - [remote_util:3654] INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2022-03-01 05:08:56,717] - [basetestcase:701] INFO - ============== basetestcase cleanup was started for test #13 suite_tearDown ============== [2022-03-01 05:08:56,742] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2022-03-01 05:08:56,750] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.106.52, nothing to delete [2022-03-01 05:08:56,756] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2022-03-01 05:08:56,763] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2022-03-01 05:08:56,768] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2022-03-01 05:08:56,773] - [bucket_helper:152] INFO - Could not find any buckets for node 172.23.105.162, nothing to delete [2022-03-01 05:08:56,773] - [basetestcase:720] INFO - Removing user 'clientuser'... [2022-03-01 05:08:56,785] - [rest_client:1073] ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2022-03-01 05:08:56,785] - [basetestcase:724] INFO - b'"User was not found."' [2022-03-01 05:08:56,791] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,810] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.104.172:8091 [2022-03-01 05:08:56,810] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,815] - [cluster_helper:94] INFO - ns_server @ 172.23.104.172:8091 is running [2022-03-01 05:08:56,821] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.106.52:8091 [2022-03-01 05:08:56,821] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,825] - [cluster_helper:94] INFO - ns_server @ 172.23.106.52:8091 is running [2022-03-01 05:08:56,828] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.158:8091 [2022-03-01 05:08:56,828] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,830] - [cluster_helper:94] INFO - ns_server @ 172.23.105.158:8091 is running [2022-03-01 05:08:56,834] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.126:8091 [2022-03-01 05:08:56,834] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,836] - [cluster_helper:94] INFO - ns_server @ 172.23.105.126:8091 is running [2022-03-01 05:08:56,839] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.127:8091 [2022-03-01 05:08:56,839] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,842] - [cluster_helper:94] INFO - ns_server @ 172.23.105.127:8091 is running [2022-03-01 05:08:56,845] - [cluster_helper:90] INFO - waiting for ns_server @ 172.23.105.162:8091 [2022-03-01 05:08:56,845] - [rest_client:44] INFO - -->is_ns_server_running? [2022-03-01 05:08:56,848] - [cluster_helper:94] INFO - ns_server @ 172.23.105.162:8091 is running [2022-03-01 05:08:56,848] - [basetestcase:743] INFO - ============== basetestcase cleanup was finished for test #13 suite_tearDown ============== [2022-03-01 05:08:56,848] - [basetestcase:755] INFO - closing all ssh connections [2022-03-01 05:08:56,849] - [basetestcase:760] INFO - closing all memcached connections *** TestRunner *** workspace is /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver fails is 0 13 Desc1: 7.1.0-2407 - os_certify nsserver - centosnonroot (13/13) python3 scripts/rerun_jobs.py 7.1.0-2407 --executor_jenkins_job --run_params=get-cbcollect-info=False,get-logs=False,get-coredumps=False INFO:merge_reports:Merging of report files from logs/**/*.xml INFO:merge_reports:-- logs/testrunner-22-Mar-01_03-12-31/report-22-Mar-01_03-12-31-rebalance.rebalancein.RebalanceInTests.xml -- INFO:merge_reports:-- logs/testrunner-22-Mar-01_03-12-31/report-22-Mar-01_03-12-31-failover.failovertests.FailoverTests.xml -- INFO:merge_reports:-- logs/testrunner-22-Mar-01_03-12-31/report-22-Mar-01_03-12-31-rebalance.rebalanceinout.RebalanceInOutTests.xml -- INFO:merge_reports:-- logs/testrunner-22-Mar-01_03-12-31/report-22-Mar-01_03-12-31-rebalance.rebalanceout.RebalanceOutTests.xml -- INFO:merge_reports:-- logs/testrunner-22-Mar-01_03-12-31/report-22-Mar-01_03-12-31-failover.AutoFailoverTests.AutoFailoverTests.xml -- INFO:merge_reports: Number of TestSuites=5 INFO:merge_reports: TestSuite#1) rebalance.rebalancein.RebalanceInTests, Number of Tests=4 INFO:merge_reports:Summary file is at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_05-08-57/merged_summary/mergedreport-22-Mar-01_05-08-57-failover.AutoFailoverTests.AutoFailoverTests.xml INFO:merge_reports: TestSuite#2) failover.failovertests.FailoverTests, Number of Tests=4 INFO:merge_reports:Summary file is at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_05-08-57/merged_summary/mergedreport-22-Mar-01_05-08-57-failover.AutoFailoverTests.AutoFailoverTests.xml INFO:merge_reports: TestSuite#3) rebalance.rebalanceinout.RebalanceInOutTests, Number of Tests=1 INFO:merge_reports:Summary file is at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_05-08-57/merged_summary/mergedreport-22-Mar-01_05-08-57-failover.AutoFailoverTests.AutoFailoverTests.xml INFO:merge_reports: TestSuite#4) rebalance.rebalanceout.RebalanceOutTests, Number of Tests=3 INFO:merge_reports:Summary file is at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_05-08-57/merged_summary/mergedreport-22-Mar-01_05-08-57-failover.AutoFailoverTests.AutoFailoverTests.xml INFO:merge_reports: TestSuite#5) failover.AutoFailoverTests.AutoFailoverTests, Number of Tests=1 INFO:merge_reports:Summary file is at /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-22-Mar-01_05-08-57/merged_summary/mergedreport-22-Mar-01_05-08-57-failover.AutoFailoverTests.AutoFailoverTests.xml summary so far suite rebalance.rebalancein.RebalanceInTests , pass 4 , fail 0 summary so far suite failover.failovertests.FailoverTests , pass 4 , fail 0 summary so far suite rebalance.rebalanceinout.RebalanceInOutTests , pass 1 , fail 0 summary so far suite rebalance.rebalanceout.RebalanceOutTests , pass 3 , fail 0 summary so far suite failover.AutoFailoverTests.AutoFailoverTests , pass 1 , fail 0 No more failed tests. Stopping reruns [description-setter] Description set: 7.1.0-2407 - os_certify nsserver - centosnonroot (13/13) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Notifying upstream projects of job completion No emails were triggered. Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Triggering a new build of post-to-gerrit Triggering a new build of test-executor-cleanup-aws Finished: SUCCESS