Started by user Balakumaran Gopal Rebuilds build #232 Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on magma-sc1401-32g-24c (magma_slave) in workspace /data/workspace/temp_rebalance_magma [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... No credentials specified Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_rebalance_magma # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision f02c53c6c15a4edffc348211647968262fab73e3 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f f02c53c6c15a4edffc348211647968262fab73e3 # timeout=10 Commit message: "Adding entry for 7.0.3 build in testconstants.py" > /usr/bin/git rev-list --no-walk bf1186b0e51d59ec3207b4957fc7d9660aa63f57 # timeout=10 [temp_rebalance_magma] $ /bin/sh -xe /tmp/jenkins2425338523639693757.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 [_1] ip:172.23.105.164 [_2] ip:172.23.105.206 [_3] ip:172.23.105.33 [_4] ip:172.23.105.36 [_5] ip:172.23.106.177 [_6] ip:172.23.122.208 [_7] ip:172.23.122.226' + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + sdk_client_params='-P transaction_version=1.1.8 -P client_version=3.1.6' + mkdir tr_for_install + cd tr_for_install + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... + cd testrunner + git checkout master Already on 'master' + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + py_executable=python + 7.1.0-2211 + grep 7 /tmp/jenkins2425338523639693757.sh: line 65: 7.1.0-2211: command not found + python3 scripts/new_install.py -i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p timeout=1200,get-cbcollect-info=True,version=7.1.0-2211,product=cb,debug_logs=False,url= 2022-02-01 23:32:42,483 - root - WARNING - URL: is not valid, will use version to locate build 2022-02-01 23:32:42,485 - root - INFO - SSH Connecting to 172.23.105.164 with username:root, attempt#1 of 5 2022-02-01 23:32:42,598 - root - INFO - SSH Connected to 172.23.105.164 as root 2022-02-01 23:32:42,892 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:43,157 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:43,160 - root - INFO - SSH Connecting to 172.23.105.206 with username:root, attempt#1 of 5 2022-02-01 23:32:43,276 - root - INFO - SSH Connected to 172.23.105.206 as root 2022-02-01 23:32:43,568 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:43,829 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:43,831 - root - INFO - SSH Connecting to 172.23.105.33 with username:root, attempt#1 of 5 2022-02-01 23:32:43,941 - root - INFO - SSH Connected to 172.23.105.33 as root 2022-02-01 23:32:44,234 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:44,497 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:44,499 - root - INFO - SSH Connecting to 172.23.105.36 with username:root, attempt#1 of 5 2022-02-01 23:32:44,603 - root - INFO - SSH Connected to 172.23.105.36 as root 2022-02-01 23:32:44,891 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:45,149 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:45,151 - root - INFO - SSH Connecting to 172.23.106.177 with username:root, attempt#1 of 5 2022-02-01 23:32:45,265 - root - INFO - SSH Connected to 172.23.106.177 as root 2022-02-01 23:32:45,575 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:45,845 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:45,848 - root - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 2022-02-01 23:32:45,967 - root - INFO - SSH Connected to 172.23.122.208 as root 2022-02-01 23:32:46,308 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:46,576 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:46,578 - root - INFO - SSH Connecting to 172.23.122.226 with username:root, attempt#1 of 5 2022-02-01 23:32:46,692 - root - INFO - SSH Connected to 172.23.122.226 as root 2022-02-01 23:32:46,994 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:47,264 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:47,266 - root - INFO - SSH Connecting to 172.23.105.164 with username:root, attempt#1 of 5 2022-02-01 23:32:47,379 - root - INFO - SSH Connected to 172.23.105.164 as root 2022-02-01 23:32:47,661 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:47,931 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:47,934 - root - INFO - SSH Connecting to 172.23.105.206 with username:root, attempt#1 of 5 2022-02-01 23:32:48,035 - root - INFO - SSH Connected to 172.23.105.206 as root 2022-02-01 23:32:48,311 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:48,568 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:48,570 - root - INFO - SSH Connecting to 172.23.105.33 with username:root, attempt#1 of 5 2022-02-01 23:32:48,672 - root - INFO - SSH Connected to 172.23.105.33 as root 2022-02-01 23:32:48,946 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:49,209 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:49,212 - root - INFO - SSH Connecting to 172.23.105.36 with username:root, attempt#1 of 5 2022-02-01 23:32:49,313 - root - INFO - SSH Connected to 172.23.105.36 as root 2022-02-01 23:32:49,588 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:49,851 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:49,855 - root - INFO - SSH Connecting to 172.23.106.177 with username:root, attempt#1 of 5 2022-02-01 23:32:49,970 - root - INFO - SSH Connected to 172.23.106.177 as root 2022-02-01 23:32:50,261 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:50,534 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:50,537 - root - INFO - SSH Connecting to 172.23.122.208 with username:root, attempt#1 of 5 2022-02-01 23:32:50,650 - root - INFO - SSH Connected to 172.23.122.208 as root 2022-02-01 23:32:50,954 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:51,216 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:51,219 - root - INFO - SSH Connecting to 172.23.122.226 with username:root, attempt#1 of 5 2022-02-01 23:32:51,328 - root - INFO - SSH Connected to 172.23.122.226 as root 2022-02-01 23:32:51,614 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2022-02-01 23:32:51,887 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2022-02-01 23:32:51,888 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,971 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:51,972 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,973 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:51,974 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,975 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:51,976 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,977 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:51,977 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,978 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:51,979 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,980 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:51,980 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:32:51,981 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/2211/couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm is live 2022-02-01 23:32:59,486 - root - INFO - Done with uninstall on 172.23.105.36. 2022-02-01 23:32:59,510 - root - INFO - Done with uninstall on 172.23.122.226. 2022-02-01 23:33:00,012 - root - INFO - Done with uninstall on 172.23.106.177. 2022-02-01 23:33:00,580 - root - INFO - Done with uninstall on 172.23.105.33. 2022-02-01 23:33:00,643 - root - INFO - Done with uninstall on 172.23.105.164. 2022-02-01 23:33:01,512 - root - INFO - Done with uninstall on 172.23.105.206. 2022-02-01 23:33:04,674 - root - INFO - Done with uninstall on 172.23.122.208. 2022-02-01 23:33:07,023 - root - INFO - running command.raw on 172.23.105.164: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,069 - root - INFO - command executed successfully with root 2022-02-01 23:33:07,134 - root - INFO - running command.raw on 172.23.105.206: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,179 - root - INFO - command executed successfully with root 2022-02-01 23:33:07,247 - root - INFO - running command.raw on 172.23.105.33: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,296 - root - INFO - command executed successfully with root 2022-02-01 23:33:07,362 - root - INFO - running command.raw on 172.23.105.36: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,408 - root - INFO - command executed successfully with root 2022-02-01 23:33:07,476 - root - INFO - running command.raw on 172.23.106.177: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,523 - root - INFO - command executed successfully with root 2022-02-01 23:33:07,590 - root - INFO - running command.raw on 172.23.122.208: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,635 - root - INFO - command executed successfully with root 2022-02-01 23:33:07,699 - root - INFO - running command.raw on 172.23.122.226: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-2211-centos7.x86_64.rpm 2022-02-01 23:33:07,744 - root - INFO - command executed successfully with root 2022-02-01 23:34:05,844 - root - INFO - Done with install on 172.23.105.36. 2022-02-01 23:34:05,844 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:05,846 - root - ERROR - socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:05,961 - root - INFO - Done with install on 172.23.105.206. 2022-02-01 23:34:05,962 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:05,964 - root - ERROR - socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:07,406 - root - INFO - Done with install on 172.23.106.177. 2022-02-01 23:34:07,407 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:07,408 - root - ERROR - socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:07,669 - root - INFO - Done with install on 172.23.105.33. 2022-02-01 23:34:07,670 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:07,671 - root - ERROR - socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:07,873 - root - INFO - Done with install on 172.23.105.164. 2022-02-01 23:34:07,873 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:07,874 - root - ERROR - socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:08,848 - root - ERROR - socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:08,968 - root - ERROR - socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:10,412 - root - ERROR - socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:10,675 - root - ERROR - socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:10,878 - root - ERROR - socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:11,855 - root - INFO - Done with install on 172.23.122.208. 2022-02-01 23:34:11,855 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:11,857 - root - ERROR - socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:14,861 - root - INFO - running command.raw on 172.23.105.36: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.36 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:14,863 - root - ERROR - socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:14,982 - root - INFO - running command.raw on 172.23.105.206: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.206 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:15,274 - root - INFO - command executed successfully with root 2022-02-01 23:34:15,278 - root - INFO - running command.raw on 172.23.105.36: rm -rf /data/* 2022-02-01 23:34:15,386 - root - INFO - command executed successfully with root 2022-02-01 23:34:15,387 - root - INFO - running command.raw on 172.23.105.36: chown -R couchbase:couchbase /data 2022-02-01 23:34:15,395 - root - INFO - command executed successfully with root 2022-02-01 23:34:15,395 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:15,426 - root - INFO - command executed successfully with root 2022-02-01 23:34:15,432 - root - INFO - running command.raw on 172.23.105.206: rm -rf /data/* 2022-02-01 23:34:15,562 - root - INFO - command executed successfully with root 2022-02-01 23:34:15,563 - root - INFO - running command.raw on 172.23.105.206: chown -R couchbase:couchbase /data 2022-02-01 23:34:15,571 - root - INFO - command executed successfully with root 2022-02-01 23:34:15,572 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:16,423 - root - INFO - running command.raw on 172.23.106.177: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.177 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:16,689 - root - INFO - running command.raw on 172.23.105.33: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.33 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:16,859 - root - INFO - command executed successfully with root 2022-02-01 23:34:16,863 - root - INFO - running command.raw on 172.23.106.177: rm -rf /data/* 2022-02-01 23:34:16,892 - root - INFO - running command.raw on 172.23.105.164: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.164 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:17,107 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,107 - root - INFO - running command.raw on 172.23.106.177: chown -R couchbase:couchbase /data 2022-02-01 23:34:17,154 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,158 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,158 - root - INFO - running command.raw on 172.23.105.33: rm -rf /data/* 2022-02-01 23:34:17,159 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:17,278 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,279 - root - INFO - running command.raw on 172.23.105.33: chown -R couchbase:couchbase /data 2022-02-01 23:34:17,287 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,288 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:17,366 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,370 - root - INFO - running command.raw on 172.23.105.164: rm -rf /data/* 2022-02-01 23:34:17,547 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,548 - root - INFO - running command.raw on 172.23.105.164: chown -R couchbase:couchbase /data 2022-02-01 23:34:17,597 - root - INFO - command executed successfully with root 2022-02-01 23:34:17,598 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:17,967 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:18,066 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:18,972 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.105.36 2022-02-01 23:34:18,973 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:18,977 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:18,977 - root - INFO - /node/controller/setupServices params on 172.23.105.36: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:19,007 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:19,007 - root - INFO - settings/web params on 172.23.105.36:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:19,068 - root - INFO - --> status:True 2022-02-01 23:34:19,068 - root - INFO - Done with init on 172.23.105.36. 2022-02-01 23:34:19,072 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.105.206 2022-02-01 23:34:19,072 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:19,076 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:19,077 - root - INFO - /node/controller/setupServices params on 172.23.105.206: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:19,078 - root - INFO - Done with cleanup on 172.23.105.36. 2022-02-01 23:34:19,107 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:19,107 - root - INFO - settings/web params on 172.23.105.206:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:19,172 - root - INFO - --> status:True 2022-02-01 23:34:19,173 - root - INFO - Done with init on 172.23.105.206. 2022-02-01 23:34:19,185 - root - INFO - Done with cleanup on 172.23.105.206. 2022-02-01 23:34:19,682 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:19,733 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:20,202 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:20,687 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.106.177 2022-02-01 23:34:20,687 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:20,692 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:20,692 - root - INFO - /node/controller/setupServices params on 172.23.106.177: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:20,722 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:20,722 - root - INFO - settings/web params on 172.23.106.177:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:20,759 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.105.33 2022-02-01 23:34:20,760 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:20,764 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:20,764 - root - INFO - /node/controller/setupServices params on 172.23.105.33: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:20,791 - root - INFO - --> status:True 2022-02-01 23:34:20,791 - root - INFO - Done with init on 172.23.106.177. 2022-02-01 23:34:20,794 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:20,794 - root - INFO - settings/web params on 172.23.105.33:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:20,805 - root - INFO - Done with cleanup on 172.23.106.177. 2022-02-01 23:34:20,855 - root - INFO - --> status:True 2022-02-01 23:34:20,855 - root - INFO - Done with init on 172.23.105.33. 2022-02-01 23:34:20,867 - root - INFO - Done with cleanup on 172.23.105.33. 2022-02-01 23:34:20,876 - root - INFO - running command.raw on 172.23.122.208: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.122.208 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:21,207 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.105.164 2022-02-01 23:34:21,207 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:21,212 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:21,212 - root - INFO - /node/controller/setupServices params on 172.23.105.164: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:21,242 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:21,242 - root - INFO - settings/web params on 172.23.105.164:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:21,308 - root - INFO - --> status:True 2022-02-01 23:34:21,309 - root - INFO - Done with init on 172.23.105.164. 2022-02-01 23:34:21,320 - root - INFO - Done with cleanup on 172.23.105.164. 2022-02-01 23:34:21,393 - root - INFO - command executed successfully with root 2022-02-01 23:34:21,397 - root - INFO - running command.raw on 172.23.122.208: rm -rf /data/* 2022-02-01 23:34:21,404 - root - INFO - command executed successfully with root 2022-02-01 23:34:21,404 - root - INFO - running command.raw on 172.23.122.208: chown -R couchbase:couchbase /data 2022-02-01 23:34:21,451 - root - INFO - command executed successfully with root 2022-02-01 23:34:21,452 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:26,773 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:27,804 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.122.208 2022-02-01 23:34:27,804 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:27,808 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:27,809 - root - INFO - /node/controller/setupServices params on 172.23.122.208: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:27,838 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:27,838 - root - INFO - settings/web params on 172.23.122.208:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:27,899 - root - INFO - --> status:True 2022-02-01 23:34:27,899 - root - INFO - Done with init on 172.23.122.208. 2022-02-01 23:34:27,909 - root - INFO - Done with cleanup on 172.23.122.208. 2022-02-01 23:34:46,195 - root - INFO - Done with install on 172.23.122.226. 2022-02-01 23:34:46,196 - root - INFO - Waiting for couchbase to be reachable 2022-02-01 23:34:46,197 - root - ERROR - socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:49,202 - root - ERROR - socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:34:55,212 - root - INFO - running command.raw on 172.23.122.226: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.122.226 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2022-02-01 23:34:55,645 - root - INFO - command executed successfully with root 2022-02-01 23:34:55,650 - root - INFO - running command.raw on 172.23.122.226: rm -rf /data/* 2022-02-01 23:34:55,657 - root - INFO - command executed successfully with root 2022-02-01 23:34:55,657 - root - INFO - running command.raw on 172.23.122.226: chown -R couchbase:couchbase /data 2022-02-01 23:34:55,706 - root - INFO - command executed successfully with root 2022-02-01 23:34:55,707 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2022-02-01 23:34:58,658 - root - INFO - Setting data_path: /data: status True 2022-02-01 23:34:59,675 - root - INFO - Setting KV memory quota as 7855 MB on 172.23.122.226 2022-02-01 23:34:59,676 - root - INFO - pools/default params : memoryQuota=7855 2022-02-01 23:34:59,680 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2022-02-01 23:34:59,680 - root - INFO - /node/controller/setupServices params on 172.23.122.226: 8091:hostname=None&user=Administrator&password=password&services=kv 2022-02-01 23:34:59,710 - root - INFO - --> in init_cluster...Administrator,password,8091 2022-02-01 23:34:59,711 - root - INFO - settings/web params on 172.23.122.226:8091:port=8091&username=Administrator&password=password 2022-02-01 23:34:59,771 - root - INFO - --> status:True 2022-02-01 23:34:59,771 - root - INFO - Done with init on 172.23.122.226. 2022-02-01 23:34:59,782 - root - INFO - Done with cleanup on 172.23.122.226. 2022-02-01 23:35:02,886 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-01 23:35:02,920 - root - INFO - cluster:C1 node:172.23.105.164:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,920 - root - INFO - cluster:C2 node:172.23.105.206:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,921 - root - INFO - cluster:C3 node:172.23.105.33:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,921 - root - INFO - cluster:C4 node:172.23.105.36:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,921 - root - INFO - cluster:C5 node:172.23.106.177:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,921 - root - INFO - cluster:C6 node:172.23.122.208:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,921 - root - INFO - cluster:C7 node:172.23.122.226:8091 version:7.1.0-2211-enterprise aFamily:inet services:['kv'] 2022-02-01 23:35:02,921 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-01 23:35:02,921 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-01 23:35:02,921 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-01 23:35:02,921 - root - INFO - INSTALL COMPLETED ON: 172.23.105.164 2022-02-01 23:35:02,921 - root - INFO - INSTALL COMPLETED ON: 172.23.105.206 2022-02-01 23:35:02,922 - root - INFO - INSTALL COMPLETED ON: 172.23.105.33 2022-02-01 23:35:02,922 - root - INFO - INSTALL COMPLETED ON: 172.23.105.36 2022-02-01 23:35:02,922 - root - INFO - INSTALL COMPLETED ON: 172.23.106.177 2022-02-01 23:35:02,922 - root - INFO - INSTALL COMPLETED ON: 172.23.122.208 2022-02-01 23:35:02,922 - root - INFO - INSTALL COMPLETED ON: 172.23.122.226 2022-02-01 23:35:02,922 - root - INFO - ---------------------------------------------------------------------------------------------------- 2022-02-01 23:35:02,922 - root - INFO - TOTAL INSTALL TIME = 140 seconds + status=0 + cd ../.. + rm -rf tr_for_install + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.8 -P client_version=3.1.6 -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -c conf/magma/dgm_collections_failover.conf -m rest' Starting a Gradle Daemon (subsequent builds will be faster) > Configure project : Using Transaction_client :: 1.1.8 Using Java_client :: 3.1.6 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_magma/build/classes/java/main:/data/workspace/temp_rebalance_magma/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.8/e69a6013e59f498f76671c1acc43df14f1163180/couchbase-transactions-1.1.8.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.6/f065d71963e08bd5577838592e33f2ca35f5d64a/java-client-3.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.6/b3ece73ab7069b1e97669500edc448163c2f4304/core-io-2.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/ssh.py -i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -c conf/magma/dgm_collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_magma/build/classes/java/main:/data/workspace/temp_rebalance_magma/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.8/e69a6013e59f498f76671c1acc43df14f1163180/couchbase-transactions-1.1.8.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.6/f065d71963e08bd5577838592e33f2ca35f5d64a/java-client-3.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.6/b3ece73ab7069b1e97669500edc448163c2f4304/core-io-2.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -c conf/magma/dgm_collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_magma/build/classes/java/main:/data/workspace/temp_rebalance_magma/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.8/e69a6013e59f498f76671c1acc43df14f1163180/couchbase-transactions-1.1.8.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.6/f065d71963e08bd5577838592e33f2ca35f5d64a/java-client-3.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.6/b3ece73ab7069b1e97669500edc448163c2f4304/core-io-2.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -c conf/magma/dgm_collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_magma/build/classes/java/main:/data/workspace/temp_rebalance_magma/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.8/e69a6013e59f498f76671c1acc43df14f1163180/couchbase-transactions-1.1.8.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.6/f065d71963e08bd5577838592e33f2ca35f5d64a/java-client-3.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.6/b3ece73ab7069b1e97669500edc448163c2f4304/core-io-2.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -c conf/magma/dgm_collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_magma/build/classes/java/main:/data/workspace/temp_rebalance_magma/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.8/e69a6013e59f498f76671c1acc43df14f1163180/couchbase-transactions-1.1.8.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.6/f065d71963e08bd5577838592e33f2ca35f5d64a/java-client-3.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.6/b3ece73ab7069b1e97669500edc448163c2f4304/core-io-2.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops-temp_rebalance_magma.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -c conf/magma/dgm_collections_failover.conf -m rest > Task :compileJava warning: unknown enum constant When.MAYBE reason: class file for javax.annotation.meta.When not found Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 warning > Task :testrunner Filename: conf/magma/dgm_collections_failover.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'disk_optimized_thread_settings': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'num_nodes': 7, 'quota_percent': '95', 'rerun': 'False', 'spec': 'dgm_collections_failover'} Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,bucket_spec=magma_dgm.10_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_rebalance_out_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_1', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_rebalance_out_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.10_percent_dgm.5_node_2_replica_magma_512', 'case_number': 1, 'randomize_value': 'True'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-01 23:35:38,255 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-01 23:35:38,933 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-01 23:35:40,207 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0x1d305df6, 0.0.0.0/0.0.0.0:59613] timeout=300.0>) 2022-02-01 23:35:43,711 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x65f5d0c1, 0.0.0.0/0.0.0.0:37981] timeout=300.0>) 2022-02-01 23:35:46,344 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x85856c93, 0.0.0.0/0.0.0.0:47209] timeout=300.0>) 2022-02-01 23:35:48,380 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x8146eda3, 0.0.0.0/0.0.0.0:54701] timeout=300.0>) 2022-02-01 23:35:50,410 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x4875d409, 0.0.0.0/0.0.0.0:45351] timeout=300.0>) 2022-02-01 23:35:55,608 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x0b32129c, 0.0.0.0/0.0.0.0:51661] timeout=300.0>) 2022-02-01 23:35:57,634 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0xf779cab5, 0.0.0.0/0.0.0.0:42369] timeout=300.0>) 2022-02-01 23:35:59,661 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xdd084826, 0.0.0.0/0.0.0.0:46293] timeout=300.0>) 2022-02-01 23:36:01,684 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0xd2d3f0e1, 0.0.0.0/0.0.0.0:52083] timeout=300.0>) 2022-02-01 23:36:07,628 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x0dd884d6, 0.0.0.0/0.0.0.0:34903] timeout=300.0>) 2022-02-01 23:36:09,648 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xc54541dd, 0.0.0.0/0.0.0.0:52767] timeout=300.0>) 2022-02-01 23:36:11,668 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x9983abf3, 0.0.0.0/0.0.0.0:43307] timeout=300.0>) 2022-02-01 23:36:13,690 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x2b9c4ef0, 0.0.0.0/0.0.0.0:60234] timeout=300.0>) 2022-02-01 23:36:19,459 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xe296fda9, 0.0.0.0/0.0.0.0:59161] timeout=300.0>) 2022-02-01 23:36:21,480 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x64617844, 0.0.0.0/0.0.0.0:38660] timeout=300.0>) 2022-02-01 23:36:23,509 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0xf228717d, 0.0.0.0/0.0.0.0:45999] timeout=300.0>) 2022-02-01 23:36:25,571 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x2b3a7d5d, 0.0.0.0/0.0.0.0:46093] timeout=300.0>) 2022-02-01 23:36:31,211 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xa16d25cc, 0.0.0.0/0.0.0.0:54463] timeout=300.0>) 2022-02-01 23:36:33,232 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0xd0299f55, 0.0.0.0/0.0.0.0:56765] timeout=300.0>) 2022-02-01 23:36:35,253 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x93cd6bfb, 0.0.0.0/0.0.0.0:48271] timeout=300.0>) 2022-02-01 23:36:37,273 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0xf5be3d82, 0.0.0.0/0.0.0.0:59037] timeout=300.0>) 2022-02-01 23:36:43,302 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x2322ae03, 0.0.0.0/0.0.0.0:52713] timeout=300.0>) 2022-02-01 23:36:45,357 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0xa72a6582, 0.0.0.0/0.0.0.0:44349] timeout=300.0>) 2022-02-01 23:36:47,408 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x78099be7, 0.0.0.0/0.0.0.0:34821] timeout=300.0>) 2022-02-01 23:36:49,447 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0x2772cfe9, 0.0.0.0/0.0.0.0:42995] timeout=300.0>) 2022-02-01 23:36:55,417 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0x8513eaa8, 0.0.0.0/0.0.0.0:37391] timeout=300.0>) 2022-02-01 23:36:57,440 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0xad3cceb3, 0.0.0.0/0.0.0.0:56423] timeout=300.0>) 2022-02-01 23:36:59,460 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0x6f271cf9, 0.0.0.0/0.0.0.0:46139] timeout=300.0>) 2022-02-01 23:37:01,479 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0xeb68a00c, 0.0.0.0/0.0.0.0:37089] timeout=300.0>) 2022-02-01 23:37:03,493 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-01 23:37:41,970 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-01 23:37:54,882 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-01 23:37:54,884 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-01 23:37:54,908 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-01 23:38:16,125 | test | INFO | pool-3-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 0.66399398647 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-01 23:38:16,161 | test | INFO | pool-3-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-01 23:38:31,220 | test | INFO | pool-3-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-01 23:38:31,250 | test | INFO | pool-3-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.128000021 sec 2022-02-01 23:38:31,407 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.313558259125 | 11.45 GiB | 10.72 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.877963125549 | 11.45 GiB | 10.71 GiB | 24.25 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 0.790166812994 | 11.45 GiB | 10.69 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.413533834586 | 11.45 GiB | 10.78 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 0.615345975135 | 11.45 GiB | 10.72 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-01 23:38:31,408 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-01 23:38:31,408 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-01 23:38:31,632 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-01 23:38:32,398 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-01 23:38:32,743 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 0.0 Byte | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ 2022-02-01 23:38:32,743 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-01 23:38:45,584 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-01 23:38:45,609 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 00:04:32,792 | test | INFO | pool-3-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****............................................................***........................******................*X | 29424.200 | | 25574.500 | .....................................................................................*****X | 25574.500 | | 9119.600 | .............................**X | 9119.600 | | 7804.500 | .........................******...*X | 8756.600 | | 8600.900 | ............................******..**X | 9157.000 | | 9137.600 | ..............................*****.***.******..........**...******.....*X | 15115.900 | | 15091.000 | ..................................................*****X | 15091.000 | | 11982.600 | .......................................*X | 11982.600 | | 11277.800 | .....................................*****X | 11277.800 | | 10020.400 | .................................*X | 10020.400 | | 9742.300 | ................................******........**....******......*X | 15142.200 | | 14679.400 | ................................................*****X | 14679.400 | | 10446.400 | ..................................**X | 10446.400 | | 9680.500 | ...............................*****........***...****X | 12680.600 | | 11114.400 | ....................................**X | 11114.400 | | 10803.800 | ...................................******...**..*******X | 12250.600 | | 11792.800 | .......................................*****X | 11792.800 | | 11259.100 | .....................................*******X | 11344.700 | | 11040.600 | ....................................*X | 11040.600 | | 10952.100 | ....................................******.*X | 11342.000 | | 11327.300 | .....................................*****X | 11327.300 | | 10017.900 | .................................*X | 10017.900 | | 9763.900 | ................................*****X | 9763.900 | | 9219.700 | ..............................********......**..******.....**X | 13206.900 | | 13031.900 | ...........................................****X | 13031.900 | | 9404.600 | ..............................**X | 9404.600 | | 8868.300 | .............................*****..***.******.*X | 10143.400 | | 10128.500 | .................................*****X | 10128.500 | | 9742.900 | ................................*X | 9742.900 | | 9510.000 | ...............................*****X | 9510.000 | | 8686.600 | ............................*X | 8686.600 | | 8596.200 | ............................*****X | 8596.200 | | 6962.100 | ......................*X | 6962.100 | | 6675.700 | .....................******..**X | 7075.100 | | 6934.200 | ......................****X | 6934.200 | | 6554.800 | .....................***.******...........**.*****X | 10495.100 | | 10302.500 | ..................................*******X | 10432.200 | | 10161.000 | .................................*X | 10161.000 | | 10098.000 | .................................*****X | 10098.000 | | 9637.100 | ...............................*X | 9637.100 | | 9089.700 | .............................*****X | 9089.700 | | 8202.400 | ..........................**.******...********..***.*****.**X | 10444.400 | | 10172.200 | .................................*****X | 10172.200 | | 7698.900 | .........................*X | 7698.900 | | 7609.700 | ........................******..........**.*****X | 10826.600 | | 10017.500 | .................................**.******.......***.****X | 12784.000 | | 11722.400 | ......................................**X | 11722.400 | | 11282.900 | .....................................****X | 11282.900 | | 9003.100 | .............................**X | 9003.100 | | 8252.400 | ...........................*****X | 8252.400 | | 8153.900 | ..........................**.******......*******X | 10277.600 | | 9089.300 | .............................*X | 9089.300 | | 8838.800 | .............................******........**...*****X | 12067.600 | | 11349.500 | .....................................*X | 11349.500 | | 11233.100 | .....................................******....*******X | 12564.300 | | 11712.700 | ......................................**X | 11712.700 | | 11691.800 | ......................................****X | 11691.800 | | 11291.900 | .....................................**X | 11291.900 | | 11139.600 | ....................................********.******.....**.*****X | 13116.900 | | 11629.800 | ......................................*X | 11629.800 | | 11419.800 | .....................................******.......**..******......**..*****X | 16308.900 | | 12172.300 | ........................................**X | 12172.300 | | 11101.300 | ....................................*****..********.***.*****X | 12072.000 | | 10459.000 | ..................................*X | 10459.000 | | 9873.500 | ................................******....**.*****X | 11392.800 | | 10605.300 | ...................................*X | 10605.300 | | 10428.800 | ..................................******.**.*****X | 10966.800 | | 10435.200 | ..................................**X | 10435.200 | | 10168.700 | .................................*****...***.****X | 11384.100 | | 9756.400 | ................................**X | 9756.400 | | 9456.000 | ...............................*****X | 9456.000 | | 9017.500 | .............................**.******.......**.*****X | 11556.600 | | 7374.300 | ........................*X | 7374.300 | | 6115.600 | ...................******........**....******...........**.*****X | 12949.300 | | 12078.900 | ........................................**X | 12078.900 | | 12035.900 | .......................................****X | 12035.900 | | 10140.100 | .................................**X | 10140.100 | | 9004.600 | .............................*****X | 9004.600 | | 5205.600 | ................**.******..........*X | 8314.400 | | 8302.400 | ...........................*****X | 8302.400 | | 5457.900 | .................*X | 5457.900 | | 4950.900 | ...............*****X | 4950.900 | | 4671.600 | ..............***.*****..........***..****X | 8463.700 | | 7556.200 | ........................**X | 7556.200 | | 7110.800 | .......................*****X | 7110.800 | | 6916.000 | ......................********......**.******......**..******....***..****X | 13038.400 | | 12862.200 | ..........................................**X | 12862.200 | | 12206.600 | ........................................****X | 12206.600 | | 9760.500 | ................................*********......*******X | 11668.200 | | 11553.500 | ......................................********..*X | 12233.200 | | 12151.900 | ........................................**************..*X | 12848.300 | | 12754.700 | ..........................................*****X | 12754.700 | | 11031.400 | ....................................*X | 11031.400 | | 10923.200 | ....................................******....***.****X | 12384.300 | | 11069.900 | ....................................**X | 11069.900 | | 10170.000 | .................................****X | 10170.000 | | 9453.200 | ...............................X | 9453.200 | | 8920.900 | .............................*X | 8920.900 | | 8873.500 | .............................*****X | 8873.500 | | 7987.200 | ..........................********.....*X | 9660.200 | | 9544.000 | ...............................******.....**...******..*X | 12642.500 | | 12462.800 | .........................................*****X | 12462.800 | | 10665.800 | ...................................**X | 10665.800 | | 9766.700 | ................................****X | 9766.700 | | 5640.200 | ..................***.*****......................***...*****X | 13278.300 | | 6241.600 | ....................*X | 6241.600 | | 4871.300 | ...............******...................**....*****X | 11662.300 | | 7720.000 | .........................*X | 7720.000 | | 7355.000 | .......................******...............**..*****X | 12205.500 | | 11102.000 | ....................................**X | 11102.000 | | 10944.300 | ....................................****X | 10944.300 | | 10628.800 | ...................................**X | 10628.800 | | 10526.700 | ..................................*****X | 10526.700 | | 9978.300 | ................................*X | 9978.300 | | 9778.400 | ................................******..*******X | 10531.800 | | 7549.900 | ........................*X | 7549.900 | | 6179.600 | ....................*****X | 6179.600 | | 5628.000 | ..................**...******............***..****X | 10734.500 | | 10490.800 | ..................................**X | 10490.800 | | 9273.200 | ..............................*****X | 9273.200 | | 5349.300 | .................*X | 5349.300 | | 5060.300 | ................******............**....******...**********X | 10796.700 | | 10474.300 | ..................................****X | 10474.300 | | 9744.000 | ................................*********..********...*******X | 11412.300 | | 11375.700 | .....................................*X | 11375.700 | | 11183.500 | .....................................******..**..******.....*******X | 14007.200 | | 11118.200 | ....................................*X | 11118.200 | | 10555.900 | ..................................******........*******X | 12907.200 | | 11173.300 | ....................................*X | 11173.300 | | 10890.600 | ....................................*****X | 10890.600 | | 10126.700 | .................................********..*X | 10615.700 | | 10199.800 | .................................******.*X | 10353.900 | | 10266.500 | .................................*****X | 10266.500 | | 8784.200 | ............................***...*****...........***.****X | 12983.400 | | 12183.200 | ........................................**X | 12183.200 | | 11949.300 | .......................................*****..***..*****X | 13129.300 | | 12491.600 | .........................................*X | 12491.600 | | 11273.000 | .....................................*****X | 11273.000 | | 10696.200 | ...................................**..*****X | 11414.800 | | 9542.700 | ...............................*X | 9542.700 | | 8921.200 | .............................******..................**.....*****X | 15677.600 | | 12195.800 | ........................................**X | 12195.800 | | 10468.400 | ..................................*****....***....*****.**X | 12961.600 | | 12660.600 | ..........................................****X | 12660.600 | | 10674.300 | ...................................**X | 10674.300 | | 9792.800 | ................................******....**..*******X | 11655.400 | | 11321.500 | .....................................*****X | 11321.500 | | 7655.000 | .........................*X | 7655.000 | | 7377.700 | ........................******.........**.******.......***..****X | 13094.000 | | 10358.700 | ..................................**X | 10358.700 | | 9459.800 | ...............................*****......***..****X | 12006.600 | | 11883.400 | .......................................**X | 11883.400 | | 11598.300 | ......................................******..**..*****X | 12843.800 | | 12705.100 | ..........................................*X | 12705.100 | | 12582.300 | .........................................****X | 12582.300 | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 00:04:33,355 | test | INFO | pool-3-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****...............................................................................**.....................******X | 1300.000 | | 263.900 | ...................*X | 263.900 | | 0.000 | ********************************************************************************..................***......*****.....................................*............**..........*****X | 1079.700 | | 282.700 | ....................*X | 282.700 | | 100.000 | ......*****X | 100.000 | | 20.300 | *X | 20.300 | | 0.000 | *******************************************************************************************************......**...******............*******X | 279.700 | | 40.600 | ..**X | 40.600 | | 0.000 | **************************************************************************************************************************.******.........................***....****X | 400.000 | | 160.900 | ...........**X | 160.900 | | 79.700 | .....*****......***...*****X | 200.000 | | 40.600 | ..*X | 40.600 | | 0.000 | ********...******............*X | 200.000 | | 159.400 | ...........*****X | 159.400 | | 0.000 | ***********************************************************.******............*******X | 179.700 | | 20.300 | *X | 20.300 | | 0.000 | ********...******............**X | 200.000 | | 159.400 | ...........****X | 159.400 | | 0.000 | **************************************************************************************************........................**......*****X | 400.000 | | 240.600 | .................*X | 240.600 | | 200.000 | ..............*****X | 200.000 | | 40.600 | ..**X | 40.600 | | 0.000 | *****..............................***........*****X | 500.000 | | 101.500 | ......*X | 101.500 | | 0.000 | ******............**...*****X | 200.000 | | 40.600 | ..**X | 40.600 | | 0.000 | **********************......**.*****X | 100.000 | | 20.300 | *X | 20.300 | | 0.000 | *********************************************************....******...................********............***.*****.......********X | 559.400 | | 81.200 | .....*X | 81.200 | | 60.900 | ...******...................*X | 300.000 | | 239.100 | .................*****X | 239.100 | | 0.000 | **********************************************************************************......**.*****X | 100.000 | | 20.300 | *X | 20.300 | | 0.000 | ******......**.........******...........................................**X | 759.400 | | 698.500 | ....................................................*****X | 698.500 | | 60.900 | ...*X | 60.900 | | 0.000 | ********.******......**********X | 100.000 | | 79.700 | .....****X | 79.700 | | 0.000 | *******************************************************************************************......**.********...******.........*X | 248.700 | | 208.100 | ...............*****X | 208.100 | | 71.600 | ....*X | 71.600 | | 51.300 | ..*****X | 51.300 | | 0.000 | *******************************************************************************************......**.*****X | 100.000 | | 20.300 | *X | 20.300 | | 0.000 | ******........................**........*****X | 420.300 | | 260.900 | ...................**X | 260.900 | | 179.700 | ............*****X | 179.700 | | 20.300 | *X | 20.300 | | 0.000 | **************************************************************X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 00:04:33,671 | test | INFO | pool-3-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****......................................................................**.................*****X | 550.000 | | 437.800 | ....................................................................X | 437.800 | | 106.700 | ...............*X | 106.700 | | 0.000 | ***********************************************************************************.*****......................................***..........*****X | 309.700 | | 148.500 | ......................*X | 148.500 | | 80.600 | ...........*****X | 80.600 | | 0.000 | *****************************************************************************************......**.*************X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ******............***...*************X | 100.000 | | 19.400 | ..*X | 19.400 | | 0.000 | ******************************************************************************************************************..******...................*******X | 136.900 | | 96.600 | ..............*******X | 100.000 | | 75.600 | ..........********X | 75.600 | | 43.800 | .....*X | 43.800 | | 24.400 | ..*****X | 24.400 | | 0.000 | **......******.........................*X | 200.000 | | 161.200 | ........................*****X | 161.200 | | 0.000 | *********************************************************......**.**************.......**.********...********X | 117.500 | | 82.500 | ............****X | 82.500 | | 1.900 | **X | 1.900 | | 0.000 | ***************************************************************************************......**..******.............**..*****X | 146.600 | | 25.000 | ..*X | 25.000 | | 0.000 | ********.******................................***.....*****X | 240.200 | | 38.900 | .....*X | 38.900 | | 0.100 | ******......**.*****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | *********************************************************************************.******......**..******.........................***.....****X | 251.900 | | 211.600 | ................................**X | 211.600 | | 190.300 | .............................****X | 190.300 | | 69.400 | ..........**X | 69.400 | | 40.300 | .....********.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ****************************************************************************************......**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ******...................**.......******................................................................***..........*****X | 630.600 | | 106.700 | ...............*X | 106.700 | | 0.000 | ******....**....******.....*X | 84.000 | | 74.300 | ..........*****X | 74.300 | | 34.700 | ....*X | 34.700 | | 25.000 | ..*****X | 25.000 | | 0.000 | ***************************..******...................**X | 138.400 | | 132.500 | ....................****X | 132.500 | | 11.600 | **X | 11.600 | | 0.000 | **********************************************..........***.....*****....**X | 122.900 | | 103.500 | ...............*****X | 103.500 | | 9.700 | ********.............*******X | 90.300 | | 9.700 | *X | 9.700 | | 0.000 | *******************************************************************************************.******......*X | 50.000 | | 40.300 | .....******...................**......*****X | 200.000 | | 38.800 | .....**X | 38.800 | | 0.000 | **********************************************************************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 00:04:34,927 | test | INFO | pool-3-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 11259.400 | ...................................................................................................X | 11259.400 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 00:04:35,188 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 29.5379116671 | 11.45 GiB | 9.03 GiB | 52.50 MiB / 3.50 GiB | 3249601 / 6532277 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 23.5602094241 | 11.45 GiB | 9.05 GiB | 24.25 MiB / 3.50 GiB | 3278142 / 6547202 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 29.7684533709 | 11.45 GiB | 9.09 GiB | 28.14 MiB / 3.50 GiB | 3204683 / 6382740 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 25.1630226314 | 11.45 GiB | 9.16 GiB | 0.0 Byte / 3.50 GiB | 3277681 / 6564507 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 27.0753512133 | 11.45 GiB | 8.93 GiB | 0.0 Byte / 3.50 GiB | 3288357 / 6551731 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2022-02-02 00:05:11,533 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 00:05:27,216 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 00:05:27,217 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 00:05:39,216 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 00:05:39,217 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 00:05:51,362 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 00:05:51,364 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 00:06:12,969 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 100000 | 9.77 GiB | 309.52 MiB | 386.01 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 496.92 MiB | 592.65 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 16287500 | 2.50 GiB | 1.50 GiB | 22.97 GiB | 28.0436776669 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 00:06:12,970 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 00:06:17,729 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_rebalance_out 2022-02-02 00:06:17,730 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_rebalance_out 2022-02-02 00:06:17,730 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:576] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 00:06:39,046 | test | INFO | pool-3-thread-3 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 21.1640000343 seconds to complete 2022-02-02 00:06:39,052 | test | INFO | pool-3-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 00:11:44,056 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 00:11:44,065 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 00:11:44,075 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 01:04:56,315 | test | INFO | pool-3-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.......................................***...................******......*X | 8850.500 | | 7589.000 | ......................................................*****X | 7589.000 | | 2283.500 | ...............*X | 2283.500 | | 1482.900 | .........*****X | 1482.900 | | 1451.500 | .........*X | 1451.500 | | 1257.900 | ........******.**.*****X | 1504.400 | | 1213.600 | .......***.******....*X | 1867.700 | | 1784.700 | ............*****X | 1784.700 | | 795.700 | ....X | 795.700 | | 795.700 | ....X | 795.700 | | 592.200 | ...*X | 592.200 | | 592.200 | ...***X | 592.200 | | 484.400 | ..*X | 484.400 | | 456.200 | ..*****X | 456.200 | | 437.500 | ..**X | 437.500 | | 429.000 | ..***X | 429.000 | | 429.000 | ..***X | 503.400 | | 487.500 | ..X | 487.500 | | 487.500 | ..*X | 487.500 | | 487.500 | ..**X | 487.500 | | 234.800 | *X | 234.800 | | 232.000 | ******...*******X | 673.300 | | 469.500 | ..X | 469.500 | | 469.500 | ..*.**X | 579.200 | | 579.200 | ...***...**.******...*X | 1523.700 | | 1461.100 | .........*****X | 1461.100 | | 1118.500 | .......**X | 1118.500 | | 1062.000 | ......*****...********X | 1440.500 | | 895.500 | .....*X | 895.500 | | 874.600 | .....*****X | 874.600 | | 767.700 | ....*X | 767.700 | | 758.500 | ....*X | 758.500 | | 758.500 | ....****....**.***************..**..******..................**.*****X | 4589.000 | | 1381.100 | .........*X | 1381.100 | | 641.500 | ...*****X | 641.500 | | 632.000 | ...**X | 632.000 | | 580.200 | ...*****..***.******........**..******...**..*****X | 3094.300 | | 2025.900 | .............*X | 2025.900 | | 1339.400 | ........*****X | 1339.400 | | 558.200 | ...*X | 558.200 | | 510.900 | ..******....***.****X | 1199.600 | | 977.800 | ......***.*****.....**X | 1893.500 | | 1643.400 | ...........*****X | 1643.400 | | 1283.600 | ........********....**..*****X | 2095.900 | | 1874.900 | ............*X | 1874.900 | | 1664.300 | ...........******....**..*****X | 2469.100 | | 1955.100 | .............**X | 1955.100 | | 1738.000 | ...........*****.....***....*****.....**X | 3638.800 | | 3571.500 | .........................******..........**..*****X | 5216.100 | | 2499.600 | .................*X | 2499.600 | | 1774.000 | ............******...........**....******....**....******.*X | 5059.200 | | 5037.500 | ...................................******.................*******X | 7264.500 | | 3561.300 | .........................*X | 3561.300 | | 2827.500 | ...................******...**.*****X | 3346.700 | | 3210.100 | ......................***..*****........**X | 4535.000 | | 4157.700 | .............................*****X | 4157.700 | | 3770.400 | ..........................**.........****X | 4988.900 | | 4988.900 | ...................................*.......................*X | 8064.100 | | 7673.800 | .......................................................*****X | 7673.800 | | 6215.600 | ............................................****X | 6233.100 | | 6233.100 | ............................................***..........**X | 7619.100 | | 7572.400 | ......................................................****X | 7572.400 | | 5069.300 | ....................................********.......**X | 6052.800 | | 5644.900 | ........................................*****X | 5644.900 | | 3239.200 | ......................*X | 3239.200 | | 3225.700 | ......................******...........................**....*X | 7465.700 | | 7465.700 | .....................................................*X | 7465.700 | | 7465.700 | .....................................................*X | 7465.700 | | 4667.900 | .................................*X | 4667.900 | | 3837.300 | ...........................*****X | 3837.300 | | 3751.000 | ..........................**....******..*X | 4577.000 | | 4106.800 | .............................*X | 4106.800 | | 4106.800 | .............................**X | 4106.800 | | 4106.800 | .............................X | 4106.800 | | 1933.100 | .............*X | 1933.100 | | 1629.800 | ..........******.........**.******..........**.....*****X | 4931.100 | | 3349.700 | .......................*X | 3349.700 | | 2745.500 | ...................******..**....******.....................**..*****X | 6739.300 | | 4025.300 | ............................*X | 4025.300 | | 3637.600 | .........................******............................***.....***X | 8174.900 | | 8174.900 | ..........................................................*..........***....****X | 10017.400 | | 7940.900 | .........................................................**X | 7940.900 | | 7622.800 | ......................................................X | 7622.800 | | 7622.800 | ......................................................*****........X | 8631.400 | | 8631.400 | ..............................................................X | 8631.400 | | 8119.400 | ..........................................................*****X | 8119.400 | | 4916.900 | ...................................*X | 4916.900 | | 4773.200 | ..................................******.........*X | 6087.400 | | 5568.800 | .......................................**X | 5568.800 | | 5568.800 | .......................................***.X | 5659.300 | | 5659.300 | ........................................*......*****X | 6495.100 | | 6103.200 | ...........................................**...**X | 6430.100 | | 6430.100 | ..............................................*X | 6430.100 | | 6430.100 | ..............................................*...............X | 8480.100 | | 8480.100 | .............................................................*X | 8480.100 | | 7698.700 | .......................................................***X | 7698.700 | | 7698.700 | .......................................................X | 7698.700 | | 4937.800 | ...................................**X | 4937.800 | | 4508.600 | ................................****X | 4508.600 | | 2683.500 | ..................**X | 2683.500 | | 2601.500 | ..................*****................***..****X | 5099.800 | | 4025.000 | ............................********...........***..*X | 5819.800 | | 5819.800 | .........................................**X | 5819.800 | | 5142.800 | ....................................**X | 5142.800 | | 4790.100 | ..................................**X | 4790.100 | | 4790.100 | ..................................*X | 4790.100 | | 3278.200 | .......................**X | 3278.200 | | 3099.900 | .....................*****....................***......*****X | 6591.200 | | 5778.800 | .........................................*X | 5778.800 | | 4990.700 | ...................................*****X | 4990.700 | | 2871.600 | ....................*X | 2871.600 | | 2552.500 | .................******.........**...******...*X | 4533.300 | | 4531.600 | ................................******.........**....*****X | 6360.600 | | 4780.600 | ..................................*X | 4780.600 | | 4129.400 | .............................*****X | 4129.400 | | 2444.700 | ................**X | 2444.700 | | 2075.300 | ..............*****X | 2075.300 | | 1450.900 | .........*X | 1450.900 | | 1170.600 | .......******....**..******....*******X | 2577.900 | | 2463.900 | .................*X | 2463.900 | | 2177.700 | ..............*****X | 2177.700 | | 1387.500 | .........**..******.............**X | 3511.000 | | 3326.400 | .......................****X | 3326.400 | | 2206.200 | ...............********..........***......***X | 4439.900 | | 4439.900 | ...............................*......**X | 5210.200 | | 4724.200 | .................................X | 4724.200 | | 4724.200 | .................................**X | 4724.200 | | 4724.200 | .................................*X | 4724.200 | | 2591.800 | ..................**.******.....................................................**..............*****X | 11943.500 | | 7808.200 | ........................................................*X | 7808.200 | | 5428.800 | ......................................*****X | 5428.800 | | 2815.500 | ...................**.*****X | 2881.800 | | 1889.300 | ............**X | 1889.300 | | 1870.200 | ............*****.................................***....****X | 6940.000 | | 3335.600 | .......................**X | 3335.600 | | 2442.200 | ................******....**...*****X | 3317.000 | | 3103.400 | .....................*X | 3103.400 | | 3007.000 | .....................*****X | 3007.000 | | 2322.700 | ................*X | 2322.700 | | 1984.700 | .............*****X | 1984.700 | | 1585.100 | ..........*X | 1585.100 | | 1482.500 | .........*****X | 1482.500 | | 1459.400 | .........******X | 1463.900 | | 1448.500 | .........***.*****.**X | 1681.400 | | 1583.300 | ..........*****X | 1583.300 | | 1423.100 | .........********.....**.*****X | 2248.000 | | 1611.700 | ..........**X | 1611.700 | | 1539.000 | ..........*****..........***...*****X | 3397.900 | | 2645.800 | ..................*X | 2645.800 | | 2617.300 | ..................******....*X | 3212.400 | | 2968.100 | ....................*****X | 2968.100 | | 2362.300 | ................*******X | 2437.800 | | 1549.700 | ..........*X | 1549.700 | | 1188.400 | .......*****X | 1188.400 | | 1037.100 | ......**X | 1037.100 | | 1010.600 | ......******....**...******.....*X | 2666.300 | | 2277.000 | ...............*****X | 2277.000 | | 1140.400 | .......********......**..******....*X | 2791.500 | | 2590.900 | .................*****X | 2590.900 | | 1737.800 | ...........**X | 1737.800 | | 1635.600 | ..........****X | 1635.600 | | 1214.600 | .......**X | 1214.600 | | 1139.000 | .......*****.***...*****................................***........**X | 7152.700 | | 7152.700 | ...................................................*X | 7152.700 | | 7152.700 | ...................................................*.......**...*****X | 8553.900 | | 6387.900 | .............................................*X | 6387.900 | | 5804.400 | .........................................*****X | 5804.400 | | 1755.300 | ...........*X | 1755.300 | | 421.100 | ..*****X | 421.100 | | 298.000 | .**X | 298.000 | | 295.000 | .*******X | 295.000 | | 292.100 | .******....**......******.........................**.........*****X | 6404.100 | | 5574.500 | .......................................*X | 5574.500 | | 4065.100 | ............................******.........**................*X | 7415.800 | | 7415.800 | .....................................................**X | 7415.800 | | 7415.800 | .....................................................*............***.****X | 9175.600 | | 7301.500 | ....................................................**X | 7301.500 | | 6535.500 | ..............................................****X | 6535.500 | | 6250.100 | ............................................**X | 6250.100 | | 5054.600 | ....................................*****..............***....****X | 7541.500 | | 4486.600 | ...............................**X | 4486.600 | | 4099.400 | .............................***X | 4099.400 | | 4099.400 | .............................*...............*X | 6257.500 | | 6257.500 | ............................................*..***X | 6419.000 | | 6419.000 | ..............................................*.......***..*****X | 7763.000 | | 4979.600 | ...................................*X | 4979.600 | | 4262.600 | ..............................*X | 4262.600 | | 4262.600 | ..............................*X | 4262.600 | | 4262.600 | ..............................**.........**...*****X | 5928.800 | | 3808.100 | ..........................*X | 3808.100 | | 3107.700 | .....................******.........................**.........******.......................**............*****X | 12454.900 | | 10900.300 | ..............................................................................*X | 10900.300 | | 9789.300 | ......................................................................******................**...*****X | 12293.800 | | 10490.800 | ...........................................................................*X | 10490.800 | | 10428.500 | ...........................................................................******....................**....*****X | 13636.600 | | 8736.700 | ...............................................................*X | 8736.700 | | 7616.300 | ......................................................*X | 7616.300 | | 7616.300 | ......................................................*X | 7616.300 | | 7616.300 | ......................................................**........*X | 8692.100 | | 8550.600 | .............................................................*****X | 8550.600 | | 7814.000 | ........................................................***.....*****................................***....****X | 13380.800 | | 7895.900 | ........................................................**X | 7895.900 | | 5989.200 | ..........................................*****X | 5989.200 | | 2761.700 | ...................*X | 2761.700 | | 2336.600 | ................******.*X | 2548.600 | | 2449.100 | ................******........**......**X | 4259.600 | | 4259.600 | ..............................***............X | 5990.200 | | 5990.200 | ..........................................X | 5990.200 | | 5725.100 | ........................................***X | 5725.100 | | 5725.100 | ........................................*X | 5725.100 | | 3383.300 | .......................**X | 3383.300 | | 3080.900 | .....................*****.**X | 3263.000 | | 3040.200 | .....................*****X | 3040.200 | | 2121.000 | ..............*X | 2121.000 | | 1999.700 | .............*****X | 1999.700 | | 1669.700 | ...........********.**...******..................................**.......******.*X | 7989.200 | | 7701.100 | .......................................................*****X | 7701.100 | | 2828.300 | ...................**X | 2828.300 | | 1592.500 | ..........*****..............***.......*****X | 4391.800 | | 4049.200 | ............................*X | 4049.200 | | 3402.600 | .......................*****X | 3402.600 | | 2671.500 | ..................**..******..*X | 3223.000 | | 2902.800 | ....................*****X | 2902.800 | | 2561.100 | .................*X | 2561.100 | | 2334.500 | ................*****X | 2334.500 | | 1513.200 | ..........**.******..........**.*****X | 3188.300 | | 2087.700 | ..............*X | 2087.700 | | 1767.200 | ...........******..**.******.....**..X | 3113.700 | | 3113.700 | .....................****X | 3113.700 | | 2269.700 | ...............*X | 2269.700 | | 1816.300 | ............*****X | 1816.300 | | 1651.800 | ...........*********.********.....**..*****X | 2834.000 | | 2640.300 | ..................*X | 2640.300 | | 2633.000 | ..................******...........***...****X | 4575.600 | | 3882.900 | ...........................**X | 3882.900 | | 3515.000 | ........................******.**.*****X | 3742.400 | | 3637.500 | .........................*X | 3637.500 | | 3468.100 | ........................*****X | 3468.100 | | 2803.300 | ...................********....********.**.******...*X | 4005.400 | | 3839.000 | ...........................*****X | 3839.000 | | 3741.300 | ..........................***.****X | 3931.000 | | 3465.800 | ........................**X | 3465.800 | | 3374.100 | .......................******................**......*****X | 6275.900 | | 5480.800 | .......................................*X | 5480.800 | | 5371.200 | ......................................*****X | 5371.200 | | 4934.400 | ...................................*X | 4934.400 | | 4004.600 | ............................*****X | 4004.600 | | 3053.700 | .....................***....*****.......***...******....................**...X | 8059.200 | | 8059.200 | ..........................................................***X | 8059.200 | | 8059.200 | ..........................................................X | 8059.200 | | 5602.600 | ........................................*X | 5602.600 | | 4893.200 | ..................................*****X | 4893.200 | | 4343.800 | ..............................**...******....*X | 5217.000 | | 4783.600 | ..................................****X | 4783.600 | | 4783.600 | ..................................X | 4783.600 | | 4411.500 | ...............................***....X | 4978.600 | | 4978.600 | ...................................****..*X | 5208.400 | | 5208.400 | .....................................*.******............*X | 7058.900 | | 6612.200 | ...............................................*****X | 6612.200 | | 4584.000 | ................................*X | 4584.000 | | 4470.500 | ...............................******.............**..........******........................**X | 10779.600 | | 9878.100 | .......................................................................****X | 9878.100 | | 6100.500 | ...........................................**X | 6100.500 | | 5158.100 | ....................................***X | 5158.100 | | 5158.100 | ....................................X | 5158.100 | | 2762.700 | ...................**X | 2762.700 | | 2597.400 | ..................******.............**......******.............................**......*****X | 9960.200 | | 9878.300 | .......................................................................**..*****X | 10181.100 | | 7929.300 | .........................................................**X | 7929.300 | | 7288.000 | ....................................................****X | 7288.000 | | 5487.800 | .......................................**X | 5487.800 | | 4805.300 | ..................................*****..................X | 7312.700 | | 7312.700 | ....................................................**......*X | 8156.400 | | 8156.400 | ..........................................................**X | 8156.400 | | 8156.400 | ..........................................................X | 8156.400 | | 6453.500 | ..............................................*X | 6453.500 | | 5847.300 | .........................................*****X | 5847.300 | | 5140.400 | ....................................**.*X | 5268.100 | | 5268.100 | .....................................****............**..*****X | 7131.200 | | 4837.300 | ..................................*X | 4837.300 | | 3699.200 | ..........................*****X | 3699.200 | | 3456.600 | ........................***...*****............***....*****X | 6036.000 | | 4713.000 | .................................*X | 4713.000 | | 4172.300 | .............................*****X | 4172.300 | | 3792.900 | ..........................*X | 3792.900 | | 3601.900 | .........................******.**X | 3711.200 | | 3472.500 | ........................*****.***.*****X | 3738.600 | | 3472.000 | ........................**.******......**....******...........**..*****X | 6777.000 | | 5465.900 | .......................................**X | 5465.900 | | 5133.100 | ....................................****X | 5133.100 | | 4435.400 | ...............................**X | 4435.400 | | 3921.300 | ...........................****X | 3921.300 | | 3847.400 | ...........................X | 3847.400 | | 3454.000 | ........................*X | 3454.000 | | 3183.000 | ......................******...**...*****X | 3993.300 | | 3296.600 | .......................**..******............***...*****...**X | 6026.500 | | 6006.400 | ...........................................*****X | 6006.400 | | 3884.300 | ...........................*X | 3884.300 | | 3378.200 | .......................******..*X | 3578.000 | | 3540.600 | ........................*****X | 3540.600 | | 2963.500 | ....................**X | 2963.500 | | 2911.900 | ....................*****.........................***........****X | 7476.800 | | 6179.200 | ............................................**X | 6179.200 | | 5482.500 | .......................................*****X | 5482.500 | | 2757.700 | ...................*X | 2757.700 | | 2250.500 | ...............******..................**......*******X | 5532.500 | | 5041.800 | ...................................*****X | 5041.800 | | 4310.900 | ..............................**X | 4310.900 | | 4156.700 | .............................****X | 4156.700 | | 4034.400 | ............................***.******....**..******..............*X | 6907.400 | | 6859.600 | .................................................*****X | 6859.600 | | 4575.800 | ................................********......**X | 5432.600 | | 4861.600 | ..................................****X | 4861.600 | | 1644.100 | ...........**X | 1644.100 | | 956.100 | ......*****X | 956.100 | | 488.800 | ..*X | 488.800 | | 450.300 | ..*****X | 450.300 | | 450.100 | ..********.***..*****.......***..******..*******X | 2448.100 | | 2164.100 | ..............**..******.**X | 2512.300 | | 2506.400 | .................*****X | 2506.400 | | 2284.500 | ...............*X | 2284.500 | | 2005.500 | .............******.**..*****X | 2328.000 | | 1502.600 | ..........*X | 1502.600 | | 1191.200 | .......*****X | 1191.200 | | 1078.500 | ......**X | 1078.500 | | 983.400 | ......*****X | 983.400 | | 427.200 | ..*X | 427.200 | | 264.300 | **************..**.******....*******X | 1209.000 | | 775.300 | ....**X | 775.300 | | 713.700 | ....*****X | 713.700 | | 694.100 | ....********..*******X | 1050.400 | | 939.700 | .....*X | 939.700 | | 939.600 | .....******..********.********X | 1337.800 | | 1232.100 | ........*X | 1232.100 | | 1149.800 | .......******.....**...*****X | 2237.500 | | 2053.200 | ..............*X | 2053.200 | | 2015.800 | .............******...**X | 2335.900 | | 2187.300 | ...............****X | 2187.300 | | 874.700 | .....**X | 874.700 | | 808.500 | ....******.........********....**.********..******..*X | 3166.800 | | 3078.500 | .....................*****X | 3078.500 | | 2532.500 | .................**.*****X | 2645.400 | | 2379.900 | ................**X | 2379.900 | | 1851.900 | ............****X | 1851.900 | | 955.800 | ......*******X | 970.500 | | 763.500 | ....**X | 763.500 | | 723.400 | ....****X | 723.400 | | 321.500 | .**X | 321.500 | | 262.000 | *********************X | 269.600 | | 260.800 | **.******.......**..******..**X | 1786.300 | | 1606.600 | ..........*****.***.******..*X | 2128.200 | | 1980.600 | .............*****X | 1980.600 | | 1795.800 | ............*******X | 1851.600 | | 1312.800 | ........*X | 1312.800 | | 1273.600 | ........*******X | 1321.500 | | 1308.900 | ........******.....**.*****X | 2048.700 | | 1645.100 | ...........**X | 1645.100 | | 1610.100 | ..........****X | 1610.100 | | 1563.600 | ..........**X | 1563.600 | | 1555.300 | ..........*****....***.******..*X | 2455.800 | | 2266.200 | ...............******...**...*****X | 3036.300 | | 2969.200 | ....................*X | 2969.200 | | 2823.600 | ...................******.............**..*****X | 4820.900 | | 4117.300 | .............................*X | 4117.300 | | 3930.100 | ...........................*****X | 3930.100 | | 1978.600 | .............**X | 1978.600 | | 1965.000 | .............*****............***...****X | 4056.100 | | 3967.300 | ............................**X | 3967.300 | | 3288.000 | .......................****X | 3288.000 | | 1883.300 | ............**X | 1883.300 | | 1840.100 | ............******..********.*X | 2263.400 | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 01:04:57,206 | test | INFO | pool-3-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...***.*****X | 16.000 | | 3.600 | *X | 3.600 | | 0.000 | ******************************************************************************************************************************************************************************************************************************************************************************.......***.*****X | 32.400 | | 5.000 | *X | 5.000 | | 0.000 | *************************************************........******..........................**X | 136.000 | | 104.000 | .........................****X | 104.000 | | 0.000 | *****************************************...***.*************X | 16.000 | | 3.600 | *X | 3.600 | | 0.000 | **************....**********X | 16.000 | | 12.400 | ..******...**...*****X | 36.000 | | 21.000 | ....*X | 21.000 | | 16.000 | ...*****X | 16.000 | | 3.600 | *X | 3.600 | | 0.000 | **************...***.*****X | 16.000 | | 4.000 | *X | 4.000 | | 0.000 | ******.........**......*******X | 59.200 | | 53.400 | ............******....**X | 68.000 | | 51.800 | ............****X | 51.800 | | 0.000 | **************************************************************************................***.....****X | 84.000 | | 32.200 | .......**X | 32.200 | | 16.000 | ...******......**...******...**......X | 87.400 | | 87.400 | .....................*X | 87.400 | | 87.400 | .....................***...*X | 100.200 | | 68.200 | ................*****X | 68.200 | | 55.800 | .............***..****X | 64.400 | | 12.200 | ..**X | 12.200 | | 3.600 | *****.......***....*****..........**X | 84.000 | | 68.200 | ................*****X | 68.200 | | 3.600 | *X | 3.600 | | 0.000 | ********.......******..........................*X | 132.400 | | 107.200 | ..........................*****X | 107.200 | | 3.600 | **.******.......********......................***.....*****X | 140.800 | | 26.600 | .....*X | 26.600 | | 0.000 | **************..........**...*******X | 52.400 | | 52.000 | ............******......................***.......****X | 168.000 | | 39.600 | .........**X | 39.600 | | 0.000 | **************...**.*****X | 16.000 | | 3.600 | **X | 3.600 | | 0.000 | ***************************************************************..........**...*****X | 52.000 | | 24.600 | .....*X | 24.600 | | 16.000 | ...*****X | 16.000 | | 3.600 | **X | 3.600 | | 0.000 | *****............***....*****X | 64.000 | | 52.000 | ............*X | 52.000 | | 48.000 | ...........*****X | 48.000 | | 10.800 | .*X | 10.800 | | 3.600 | ******.......*******X | 28.400 | | 3.600 | ********....**X | 16.000 | | 12.400 | ..****X | 12.400 | | 0.000 | *************************......**..*****X | 32.000 | | 7.200 | *X | 7.200 | | 0.000 | ***********************.......................................**.............*****X | 204.000 | | 154.400 | ......................................*X | 154.400 | | 140.000 | ..................................*****X | 140.000 | | 46.600 | ..........**X | 46.600 | | 24.600 | .....*****.......**X | 51.600 | | 43.400 | ..........*****X | 43.400 | | 4.000 | *X | 4.000 | | 0.000 | ********************************************************************************......**..*****X | 32.000 | | 7.200 | **........******.......................................*X | 191.400 | | 167.600 | .........................................*****X | 167.600 | | 12.600 | ..**X | 12.600 | | 0.000 | ******************************......***........*****..............**X | 111.200 | | 78.800 | ...................*****X | 78.800 | | 39.400 | .........**...*****X | 52.000 | | 12.600 | ..*X | 12.600 | | 7.200 | ******.......**.******.......********...********.***.**X | 84.600 | | 84.600 | ....................**.......**X | 111.200 | | 110.800 | ...........................****X | 110.800 | | 84.200 | ....................**X | 84.200 | | 84.000 | ....................*****X | 84.000 | | 19.800 | ....*X | 19.800 | | 0.000 | **************************************...***.****X | 16.000 | | 4.000 | **X | 4.000 | | 0.000 | **************.....................X | 82.800 | | 82.800 | ....................*....*****X | 100.400 | | 21.200 | ....*X | 21.200 | | 9.000 | .******.......*X | 36.000 | | 27.000 | .....******......................................................................***........................****X | 392.000 | | 160.000 | .......................................**X | 160.000 | | 91.600 | ......................*****X | 91.600 | | 37.600 | ........*X | 37.600 | | 12.400 | ..******......**.....******.*X | 58.800 | | 43.400 | ..........*****X | 43.400 | | 4.000 | **X | 4.000 | | 0.000 | *****...***..*****X | 20.000 | | 5.000 | **....******........................*X | 115.000 | | 102.400 | .........................*****X | 102.400 | | 9.000 | .**X | 9.000 | | 0.000 | **********************....................................**.............*****X | 192.600 | | 91.200 | ......................**X | 91.200 | | 61.400 | ..............*****......**X | 83.600 | | 62.000 | ..............*****X | 62.000 | | 0.000 | ********...**.*****X | 16.000 | | 3.600 | *X | 3.600 | | 0.000 | *********************************************************************************************************...**.******......**..*****X | 48.000 | | 23.200 | ....**X | 23.200 | | 21.000 | ....******.*X | 23.600 | | 15.000 | ..*****X | 15.000 | | 11.100 | .**......******..............*X | 89.300 | | 71.700 | .................***X | 71.700 | | 71.700 | .................*X | 71.700 | | 68.200 | ................**....*********.****X | 88.000 | | 21.600 | ....**X | 21.600 | | 0.000 | **************...**.******...................**......*****X | 116.000 | | 27.400 | .....*X | 27.400 | | 0.000 | *****************.******....********.............................**......*****X | 160.400 | | 32.800 | .......*X | 32.800 | | 4.000 | ******........................***.....****X | 120.000 | | 50.200 | ...........**X | 50.200 | | 32.000 | .......*****X | 32.000 | | 7.600 | *X | 7.600 | | 0.000 | ******..............................**..........*****X | 160.000 | | 38.800 | ........*X | 38.800 | | 0.000 | ***************...**..*****X | 20.000 | | 5.000 | *X | 5.000 | | 0.000 | ***************************************...**.*****X | 16.000 | | 4.000 | *X | 4.000 | | 3.600 | ******.......********..............***......*****X | 108.000 | | 26.600 | .....*X | 26.600 | | 4.000 | ******......*X | 28.400 | | 28.000 | ......*****X | 28.000 | | 3.600 | **X | 3.600 | | 0.000 | ***************************************************************......***.....*****.............**X | 94.200 | | 91.400 | ......................*****X | 91.400 | | 14.800 | ..*X | 14.800 | | 0.000 | *******************************...**.******...**.*****X | 32.000 | | 7.200 | **X | 7.200 | | 5.000 | ******....*X | 20.000 | | 15.000 | ..******.....................**........*****X | 128.000 | | 31.200 | ......**X | 31.200 | | 0.000 | ********.*****...***..******......**...******.........................*X | 158.400 | | 157.400 | .......................................*****X | 157.400 | | 21.600 | ....*X | 21.600 | | 0.000 | ************************************************.................................**...........*****X | 176.000 | | 43.200 | ..........*X | 43.200 | | 0.000 | *********************************..******......**X | 32.000 | | 24.000 | .....*****X | 24.000 | | 0.000 | ****************...**..*****X | 20.000 | | 5.000 | **X | 5.000 | | 0.000 | ****************************************************************...**.*****X | 16.000 | | 4.000 | *X | 4.000 | | 0.000 | ***********************...**.*****X | 16.000 | | 4.000 | **X | 4.000 | | 0.000 | *****.............***....*****X | 68.000 | | 17.000 | ...*X | 17.000 | | 0.000 | ****************..******......**X | 32.000 | | 24.000 | .....****X | 24.000 | | 12.000 | ..***.*****X | 16.000 | | 4.000 | *X | 4.000 | | 0.000 | ************************************************************************************************.........**...*****X | 48.000 | | 11.600 | .*X | 11.600 | | 0.000 | ******.......**...*****X | 40.000 | | 25.000 | .....**X | 25.000 | | 20.000 | ....*****.........***...*****X | 68.000 | | 17.000 | ...*X | 17.000 | | 0.000 | ******................**.....*****X | 84.000 | | 21.000 | ....*X | 21.000 | | 0.000 | **************************************************************************************************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 01:04:57,244 | test | INFO | pool-3-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **......***...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ************************************************************************......**...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ***********************......**......*******X | 10.500 | | 8.000 | ........*******X | 8.000 | | 5.500 | .....*****X | 5.500 | | 0.000 | *********************************......***...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ***********************************************............................***............****X | 34.000 | | 26.200 | .............................X | 26.200 | | 9.600 | ..........*X | 9.600 | | 0.000 | *********************************..******.......*X | 8.000 | | 5.500 | .....******.................................**.................*****X | 48.000 | | 20.500 | .......................**X | 20.500 | | 8.000 | ........*****X | 8.000 | | 2.500 | .********.............**.........******.......*X | 26.500 | | 16.500 | ..................*****X | 16.500 | | 0.000 | ********......***.........******......*X | 18.500 | | 11.000 | ............*****X | 11.000 | | 5.500 | .....**...********........******.......................*X | 34.000 | | 26.900 | ..............................******....**............*****X | 40.000 | | 20.400 | .......................**X | 20.400 | | 10.000 | ..........*****.......***......****X | 20.500 | | 12.600 | .............**X | 12.600 | | 5.500 | .....*****X | 5.500 | | 0.000 | ********...............**..............******.......................*X | 44.000 | | 36.900 | ..........................................*****X | 36.900 | | 4.600 | ....**X | 4.600 | | 0.000 | ******.............**.....******....................**.........******......................X | 58.900 | | 58.900 | ....................................................................*.........*****X | 66.000 | | 19.600 | ......................**X | 19.600 | | 2.500 | .*****....................***...*****X | 21.500 | | 5.000 | ....*X | 5.000 | | 0.000 | ******................................................**.......................***X | 60.500 | | 60.500 | ......................................................................*X | 60.500 | | 25.100 | ............................*X | 25.100 | | 15.100 | ................******.......................*X | 34.000 | | 24.400 | ...........................*****X | 24.400 | | 0.000 | ********......***...*****.........***...****X | 18.000 | | 4.600 | ....*X | 4.600 | | 0.000 | *****************..******.............................**......*****X | 31.500 | | 7.100 | .......*X | 7.100 | | 0.000 | ******......**......******.............********X | 21.500 | | 16.000 | .................********..........................**........******.........**.....................**X | 69.600 | | 69.600 | .................................................................................*X | 69.600 | | 69.600 | .................................................................................X | 69.600 | | 59.600 | .....................................................................*X | 59.600 | | 30.400 | ..................................*****X | 30.400 | | 7.900 | ........***..****X | 10.000 | | 2.100 | .**X | 2.100 | | 0.000 | ******************************...................***.........****X | 24.000 | | 7.500 | .......**X | 7.500 | | 0.000 | ********........*****......................***.......******..........*X | 40.000 | | 27.500 | ...............................*****X | 27.500 | | 11.000 | ............**.....*****X | 16.000 | | 12.900 | ..............*X | 12.900 | | 10.000 | ..........******.................................................***....................*****X | 68.000 | | 35.700 | .........................................*X | 35.700 | | 24.000 | ...........................*****X | 24.000 | | 7.500 | .......*X | 7.500 | | 0.000 | *******************************.............**...........*******X | 21.000 | | 11.000 | ............*****X | 11.000 | | 0.000 | **..******.............*******X | 13.500 | | 8.000 | ........*********.............................**...........*****X | 42.000 | | 17.600 | ...................*X | 17.600 | | 10.500 | ...........********..******.........................................................**.........*****X | 67.800 | | 25.200 | ............................*X | 25.200 | | 13.500 | ..............*****X | 13.500 | | 2.500 | .*X | 2.500 | | 0.000 | **************......***...********........******...................*X | 30.500 | | 21.300 | ........................*****X | 21.300 | | 0.000 | *****************************************............................**............*****X | 34.000 | | 9.600 | ..........*X | 9.600 | | 0.000 | *****************..******.......*X | 7.700 | | 5.800 | .....********...******................***...........****X | 30.600 | | 25.100 | ............................**X | 25.100 | | 13.400 | ..............*****X | 13.400 | | 0.000 | *****************************************......**...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ***************************************......**...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ********************************************************......**...........******................*X | 28.500 | | 18.900 | .....................*****X | 18.900 | | 0.000 | **..******.............***...****X | 15.600 | | 12.500 | .............**X | 12.500 | | 7.900 | ........*****X | 7.900 | | 0.000 | ********...................................................................**.........................*****X | 78.000 | | 72.500 | ....................................................................................*X | 72.500 | | 70.000 | .................................................................................*****X | 70.000 | | 43.200 | ..................................................**X | 43.200 | | 34.000 | .......................................*X | 34.000 | | 34.000 | .......................................**X | 34.000 | | 9.600 | ..........**X | 9.600 | | 0.000 | **************...............**......******..........................**............*****X | 50.000 | | 14.600 | ................*X | 14.600 | | 0.000 | **************............................................**..........................*****X | 59.500 | | 38.200 | ............................................*X | 38.200 | | 16.500 | ..................*****X | 16.500 | | 0.000 | *****************.........................................................**.......................X | 68.000 | | 68.000 | ...............................................................................***X | 68.000 | | 68.000 | ...............................................................................X | 68.000 | | 19.200 | .....................*X | 19.200 | | 0.000 | ******.........**..*****X | 10.000 | | 2.100 | .********.......**X | 8.000 | | 5.500 | .....******.........**...................******......................................*X | 61.600 | | 50.700 | ..........................................................*****X | 50.700 | | 5.000 | ....********.................................***...*****X | 35.000 | | 7.500 | .......*X | 7.500 | | 0.000 | ******************************............................***............*****X | 34.000 | | 9.600 | ..........*X | 9.600 | | 0.000 | ********..******......................***....****X | 23.900 | | 5.000 | ....**X | 5.000 | | 0.000 | **************......**...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ********.......******..........................**........******.................................**X | 63.000 | | 53.800 | ..............................................................****X | 53.800 | | 5.000 | ....**X | 5.000 | | 2.500 | .******.............*******X | 13.500 | | 2.500 | .*X | 2.500 | | 0.000 | ******.........................**........*****X | 28.000 | | 14.600 | ................**X | 14.600 | | 10.000 | ..........*****X | 10.000 | | 2.100 | .********.............................**......*****X | 31.500 | | 7.100 | .......**X | 7.100 | | 0.000 | *****.............***.....******...*X | 18.400 | | 18.000 | ....................*****X | 18.000 | | 4.600 | ....********..........................................**......*X | 45.000 | | 45.000 | ....................................................*X | 45.000 | | 45.000 | ....................................................*X | 45.000 | | 15.100 | ................*X | 15.100 | | 8.000 | ........*****X | 8.000 | | 2.500 | .**X | 2.500 | | 0.000 | **********************......................**........*****X | 26.000 | | 12.600 | .............**............******......................................*X | 54.500 | | 40.300 | ..............................................*****X | 40.300 | | 8.000 | ........**X | 8.000 | | 5.500 | .....****X | 5.500 | | 1.400 | X | 1.400 | | 0.000 | **..............******.......................................*X | 45.500 | | 37.600 | ...........................................*****X | 37.600 | | 25.900 | .............................*******X | 25.900 | | 23.500 | ..........................**...******....................***.........***X | 50.000 | | 50.000 | ..........................................................X | 50.000 | | 14.600 | ................**X | 14.600 | | 0.000 | **************............................................**.................*****X | 52.000 | | 14.200 | ...............**X | 14.200 | | 0.000 | **************.............**.....*****X | 16.000 | | 10.500 | ...........*X | 10.500 | | 8.000 | ........******...**X | 10.400 | | 10.000 | ..........****X | 10.000 | | 7.600 | .......***.******.............................................**................*****X | 60.000 | | 22.200 | .........................*X | 22.200 | | 8.000 | ........*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ********....******...................................*******X | 33.800 | | 4.600 | ....**X | 4.600 | | 0.000 | ******......**...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | *****************..******.......*X | 8.000 | | 5.500 | .....******...................................**..................*****X | 50.000 | | 14.600 | ................*X | 14.600 | | 0.000 | ***************.........**..*****X | 10.000 | | 2.100 | .*X | 2.100 | | 0.000 | ******................**.........******..........................................................***.................*****X | 84.600 | | 20.900 | .......................*X | 20.900 | | 0.000 | ******............................**..................******.*X | 40.300 | | 24.400 | ...........................*****X | 24.400 | | 5.500 | .....***...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ******.........................**........*****X | 28.000 | | 6.700 | ......*X | 6.700 | | 2.500 | .******.......**X | 8.000 | | 5.500 | .....******...................**........*****X | 28.000 | | 14.600 | ................*X | 14.600 | | 10.000 | ..........*****X | 10.000 | | 2.100 | .**X | 2.100 | | 0.000 | *****.............***........*****X | 18.500 | | 13.000 | ..............*X | 13.000 | | 8.000 | ........*******X | 8.000 | | 5.500 | .....*****X | 5.500 | | 0.000 | ********......***...****X | 8.000 | | 2.500 | .**X | 2.500 | | 0.000 | ******************************......................***........*****X | 26.000 | | 7.100 | .......*X | 7.100 | | 0.000 | *********************************.....******.............**X | 16.000 | | 11.000 | ............****X | 11.000 | | 0.000 | *********..................**.....*****X | 20.000 | | 4.200 | ...*X | 4.200 | | 0.000 | **********************************************************************************************************.**...******....*X | 7.100 | | 4.600 | ....******..........***......****X | 18.000 | | 4.600 | ....**X | 4.600 | | 0.000 | ***************************************....**...*****X | 6.700 | | 3.800 | ...*X | 3.800 | | 1.300 | *****X | 1.300 | | 0.000 | *****************************************************************...............***......****X | 18.000 | | 4.600 | ....**X | 4.600 | | 0.000 | ******......**...*****X | 8.000 | | 2.500 | .*X | 2.500 | | 0.000 | ******.........**..******..........***.......****X | 24.000 | | 13.000 | ..............**X | 13.000 | | 8.000 | ........****X | 8.000 | | 2.500 | .**X | 2.500 | | 0.000 | ************************..******.......*X | 8.000 | | 5.500 | .....*********...****X | 8.000 | | 2.500 | .**X | 2.500 | | 0.000 | *****************************************************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 01:04:58,545 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------+----------+ | 2310.700 | ...................................................................................................X | 2310.700 | +----------+------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 01:04:59,940 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 01:04:59,969 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 01:24:12,279 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 01:24:12,288 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 01:24:12,387 | test | INFO | pool-3-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 4.18236623964 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 4.54431333166 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 4.02558314522 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.425745053844 | --- OUT ---> | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 5.99472162875 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 01:24:12,407 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 01:24:22,441 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 5.03 2022-02-02 01:24:32,484 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 5.19 2022-02-02 01:24:42,525 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 7.66 2022-02-02 01:24:52,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 12.28 2022-02-02 01:25:02,611 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 13.47 2022-02-02 01:25:12,665 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 15.91 2022-02-02 01:25:22,713 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 18.63 2022-02-02 01:25:32,765 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 19.83 2022-02-02 01:25:42,808 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 21.09 2022-02-02 01:25:52,846 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 22.66 2022-02-02 01:26:02,888 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 25.07 2022-02-02 01:26:12,948 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 27.25 2022-02-02 01:26:22,996 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 29.26 2022-02-02 01:26:33,053 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 30.29 2022-02-02 01:26:43,099 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 31.05 2022-02-02 01:26:53,170 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 31.87 2022-02-02 01:27:03,273 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 31.92 2022-02-02 01:27:13,319 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 32.19 2022-02-02 01:27:23,362 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 32.62 2022-02-02 01:27:33,405 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 33.06 2022-02-02 01:27:43,451 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 34.04 2022-02-02 01:27:53,516 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 36.01 2022-02-02 01:28:03,575 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 36.5 2022-02-02 01:28:13,621 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 36.88 2022-02-02 01:28:23,664 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 37.59 2022-02-02 01:28:33,707 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 38.73 2022-02-02 01:28:43,750 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 40.13 2022-02-02 01:28:53,795 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 41.22 2022-02-02 01:29:03,838 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 42.09 2022-02-02 01:29:13,882 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 42.69 2022-02-02 01:29:23,930 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 43.35 2022-02-02 01:29:33,990 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 44.28 2022-02-02 01:29:44,063 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 44.88 2022-02-02 01:29:54,115 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:30:04,170 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:30:14,267 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:30:24,309 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:30:34,352 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:30:44,401 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:30:54,457 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:31:04,496 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:31:14,549 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:31:24,598 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:31:34,640 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:31:44,681 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:31:54,724 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:32:04,763 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:32:14,806 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:32:24,854 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:32:34,897 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:32:44,937 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.56 2022-02-02 01:32:55,003 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.62 2022-02-02 01:33:05,051 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 46.62 2022-02-02 01:33:15,092 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 48.35 2022-02-02 01:33:25,131 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 49.38 2022-02-02 01:33:35,173 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 49.65 2022-02-02 01:33:45,214 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 49.87 2022-02-02 01:33:55,253 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 50.09 2022-02-02 01:34:05,298 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 50.59 2022-02-02 01:34:15,342 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 51.14 2022-02-02 01:34:25,384 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 52.67 2022-02-02 01:34:35,457 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 53.17 2022-02-02 01:34:45,535 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 53.17 2022-02-02 01:34:55,578 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 53.17 2022-02-02 01:35:05,618 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 53.38 2022-02-02 01:35:15,667 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 54.25 2022-02-02 01:35:25,717 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 55.16 2022-02-02 01:35:35,759 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 55.71 2022-02-02 01:35:45,799 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 56.52 2022-02-02 01:35:55,851 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 57.33 2022-02-02 01:36:05,890 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 58.68 2022-02-02 01:36:15,930 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 60.96 2022-02-02 01:36:25,967 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 61.71 2022-02-02 01:36:36,013 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 63.0 2022-02-02 01:36:46,059 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 64.62 2022-02-02 01:36:56,098 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 65.7 2022-02-02 01:37:06,160 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 66.56 2022-02-02 01:37:16,230 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 66.88 2022-02-02 01:37:26,299 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 01:37:36,378 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 67.69 2022-02-02 01:37:46,438 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 68.02 2022-02-02 01:37:56,487 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 68.34 2022-02-02 01:38:06,552 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 68.56 2022-02-02 01:38:16,601 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 68.83 2022-02-02 01:38:26,651 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.1 2022-02-02 01:38:36,694 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.2 2022-02-02 01:38:46,766 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.31 2022-02-02 01:38:56,809 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.37 2022-02-02 01:39:06,848 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.53 2022-02-02 01:39:16,885 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.69 2022-02-02 01:39:26,927 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.85 2022-02-02 01:39:36,970 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.85 2022-02-02 01:39:47,025 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 69.96 2022-02-02 01:39:57,098 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.12 2022-02-02 01:40:07,154 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.23 2022-02-02 01:40:17,200 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.45 2022-02-02 01:40:27,250 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.67 2022-02-02 01:40:37,290 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.72 2022-02-02 01:40:47,335 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.77 2022-02-02 01:40:57,378 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 70.94 2022-02-02 01:41:07,418 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.1 2022-02-02 01:41:17,463 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.21 2022-02-02 01:41:27,513 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.32 2022-02-02 01:41:37,559 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.43 2022-02-02 01:41:47,601 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.59 2022-02-02 01:41:57,644 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.91 2022-02-02 01:42:07,694 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 71.97 2022-02-02 01:42:17,763 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.02 2022-02-02 01:42:27,813 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.13 2022-02-02 01:42:37,878 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.46 2022-02-02 01:42:47,948 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.57 2022-02-02 01:42:57,993 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.57 2022-02-02 01:43:08,066 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.68 2022-02-02 01:43:18,117 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.78 2022-02-02 01:43:28,174 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 72.89 2022-02-02 01:43:38,220 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 73.0 2022-02-02 01:43:48,262 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 73.0 2022-02-02 01:43:58,303 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 73.27 2022-02-02 01:44:08,349 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 73.44 2022-02-02 01:44:18,398 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 73.65 2022-02-02 01:44:28,454 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 74.14 2022-02-02 01:44:38,520 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 74.41 2022-02-02 01:44:48,608 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 74.85 2022-02-02 01:44:58,750 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.01 2022-02-02 01:45:08,808 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.12 2022-02-02 01:45:18,855 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.22 2022-02-02 01:45:28,891 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.33 2022-02-02 01:45:38,933 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.33 2022-02-02 01:45:48,973 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.38 2022-02-02 01:45:59,023 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.49 2022-02-02 01:46:09,065 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.71 2022-02-02 01:46:19,105 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 75.76 2022-02-02 01:46:29,145 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.03 2022-02-02 01:46:39,184 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.2 2022-02-02 01:46:49,226 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.2 2022-02-02 01:46:59,266 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.2 2022-02-02 01:47:09,306 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.25 2022-02-02 01:47:19,365 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.25 2022-02-02 01:47:29,407 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.3 2022-02-02 01:47:39,451 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.3 2022-02-02 01:47:49,493 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.3 2022-02-02 01:47:59,538 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.36 2022-02-02 01:48:09,579 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.47 2022-02-02 01:48:19,621 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.47 2022-02-02 01:48:29,665 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.47 2022-02-02 01:48:39,709 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.47 2022-02-02 01:48:49,749 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.57 2022-02-02 01:48:59,793 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.57 2022-02-02 01:49:09,838 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.63 2022-02-02 01:49:19,881 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.63 2022-02-02 01:49:29,921 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:49:39,964 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:49:50,016 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:50:00,062 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:50:10,111 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:50:20,154 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:50:30,197 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:50:40,243 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:50:50,280 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:51:00,321 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:51:10,364 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:51:20,404 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:51:30,444 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:51:40,483 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.74 2022-02-02 01:51:50,526 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.79 2022-02-02 01:52:00,575 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.79 2022-02-02 01:52:10,634 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.79 2022-02-02 01:52:20,683 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.79 2022-02-02 01:52:30,726 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.79 2022-02-02 01:52:40,763 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:52:50,802 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:53:00,842 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:53:10,882 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:53:20,921 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:53:30,966 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:53:41,010 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:53:51,052 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:54:01,096 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:54:11,141 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:54:21,184 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:54:31,226 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 76.9 2022-02-02 01:54:41,270 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.01 2022-02-02 01:54:51,312 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.01 2022-02-02 01:55:01,352 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.06 2022-02-02 01:55:11,397 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:55:21,441 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:55:31,483 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:55:41,528 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:55:51,575 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:56:01,618 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:56:11,658 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:56:21,703 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:56:31,749 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:56:41,789 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:56:51,836 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:57:01,890 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:57:11,930 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:57:21,971 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:57:32,016 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:57:42,059 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.11 2022-02-02 01:57:52,111 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:58:02,176 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:58:12,223 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:58:22,266 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:58:32,319 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:58:42,361 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:58:52,428 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:59:02,476 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:59:12,520 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 01:59:22,568 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.28 2022-02-02 01:59:32,611 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.28 2022-02-02 01:59:42,657 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.28 2022-02-02 01:59:52,698 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.33 2022-02-02 02:00:02,746 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:00:12,796 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:00:22,845 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:00:32,894 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:00:42,960 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:00:53,007 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:01:03,058 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:01:13,108 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:01:23,158 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:01:33,213 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:01:43,263 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:01:53,303 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:02:03,345 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:02:13,394 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:02:23,438 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.38 2022-02-02 02:02:33,486 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:02:43,536 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:02:53,582 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:03:03,627 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:03:13,674 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:03:23,720 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:03:33,759 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:03:43,802 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 02:03:53,846 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.49 2022-02-02 02:04:03,895 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.49 2022-02-02 02:04:13,944 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.49 2022-02-02 02:04:23,983 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.49 2022-02-02 02:04:34,020 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.49 2022-02-02 02:04:44,066 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.55 2022-02-02 02:04:54,111 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.55 2022-02-02 02:05:04,161 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.6 2022-02-02 02:05:14,206 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.6 2022-02-02 02:05:24,249 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.6 2022-02-02 02:05:34,289 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.6 2022-02-02 02:05:44,335 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.66 2022-02-02 02:05:54,381 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.66 2022-02-02 02:06:04,421 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.66 2022-02-02 02:06:14,467 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:06:24,506 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:06:34,548 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:06:44,596 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:06:54,648 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:07:04,726 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:07:14,802 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:07:24,859 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:07:34,911 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:07:44,957 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:07:55,003 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:08:05,046 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:08:15,092 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:08:25,131 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:08:35,177 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:08:45,220 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:08:55,267 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.71 2022-02-02 02:09:05,312 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:09:15,355 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:09:25,398 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:09:35,440 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:09:45,483 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:09:55,526 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:10:05,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:10:15,608 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:10:25,655 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:10:35,691 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.77 2022-02-02 02:10:45,730 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.99 2022-02-02 02:10:55,775 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.99 2022-02-02 02:11:05,818 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.99 2022-02-02 02:11:15,858 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 77.99 2022-02-02 02:11:25,900 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:11:35,941 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:11:45,986 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:11:56,026 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:12:06,072 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:12:16,115 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:12:26,151 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:12:36,194 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:12:46,240 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:12:56,282 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:13:06,326 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:13:16,368 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:13:26,411 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.04 2022-02-02 02:13:36,450 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:13:46,490 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:13:56,529 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:14:06,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:14:16,609 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:14:26,651 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:14:36,688 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:14:46,730 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.09 2022-02-02 02:14:56,776 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.2 2022-02-02 02:15:06,815 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.2 2022-02-02 02:15:16,848 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.2 2022-02-02 02:15:26,891 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.2 2022-02-02 02:15:36,926 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.26 2022-02-02 02:15:46,976 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.26 2022-02-02 02:15:57,016 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.26 2022-02-02 02:16:07,049 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:16:17,088 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:16:27,122 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:16:37,160 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:16:47,197 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:16:57,239 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:17:07,279 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.31 2022-02-02 02:17:17,321 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.37 2022-02-02 02:17:27,361 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.42 2022-02-02 02:17:37,404 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.48 2022-02-02 02:17:47,446 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.48 2022-02-02 02:17:57,484 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.48 2022-02-02 02:18:07,529 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.53 2022-02-02 02:18:17,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.53 2022-02-02 02:18:27,605 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.53 2022-02-02 02:18:37,645 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.58 2022-02-02 02:18:47,688 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.64 2022-02-02 02:18:57,729 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.64 2022-02-02 02:19:07,769 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.69 2022-02-02 02:19:17,806 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.75 2022-02-02 02:19:27,846 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.75 2022-02-02 02:19:37,894 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.75 2022-02-02 02:19:47,937 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.8 2022-02-02 02:19:57,976 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.86 2022-02-02 02:20:08,022 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.91 2022-02-02 02:20:18,130 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:20:28,214 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:20:38,262 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:20:48,311 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:20:58,357 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:21:08,424 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:21:18,470 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:21:28,513 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:21:38,562 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:21:48,611 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:21:58,654 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:22:08,693 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:22:18,740 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:22:28,779 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:22:38,816 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:22:48,852 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:22:58,898 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:23:08,940 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:23:18,986 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:23:29,028 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:23:39,078 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:23:49,128 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:23:59,204 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 78.96 2022-02-02 02:24:09,289 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.07 2022-02-02 02:24:19,352 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.07 2022-02-02 02:24:29,404 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.13 2022-02-02 02:24:39,456 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.13 2022-02-02 02:24:49,499 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.13 2022-02-02 02:24:59,540 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:25:09,589 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:25:19,634 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:25:29,677 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:25:39,720 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:25:49,763 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:25:59,812 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:26:09,859 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.18 2022-02-02 02:26:19,904 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.24 2022-02-02 02:26:29,946 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.24 2022-02-02 02:26:39,987 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.24 2022-02-02 02:26:50,023 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.24 2022-02-02 02:27:00,062 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.24 2022-02-02 02:27:10,101 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.24 2022-02-02 02:27:20,138 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.34 2022-02-02 02:27:30,174 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.34 2022-02-02 02:27:40,217 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:27:50,256 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:28:00,295 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:28:10,345 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:28:20,388 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:28:30,424 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:28:40,469 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:28:50,513 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:29:00,558 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:29:10,601 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.4 2022-02-02 02:29:20,642 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:29:30,686 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:29:40,729 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:29:50,769 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:30:00,811 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:30:10,848 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:30:20,888 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:30:30,937 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:30:40,982 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:30:51,019 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:31:01,059 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:31:11,102 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:31:21,153 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:31:31,193 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:31:41,236 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:31:51,289 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.45 2022-02-02 02:32:01,342 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.51 2022-02-02 02:32:11,388 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.51 2022-02-02 02:32:21,446 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.51 2022-02-02 02:32:31,493 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.51 2022-02-02 02:32:41,540 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:32:51,582 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:33:01,625 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:33:11,670 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:33:21,707 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:33:31,753 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:33:41,799 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:33:51,844 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:34:01,884 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.62 2022-02-02 02:34:11,928 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.67 2022-02-02 02:34:21,973 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.67 2022-02-02 02:34:32,016 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.67 2022-02-02 02:34:42,068 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.72 2022-02-02 02:34:52,111 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.72 2022-02-02 02:35:02,148 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.72 2022-02-02 02:35:12,187 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.72 2022-02-02 02:35:22,224 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.72 2022-02-02 02:35:32,263 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.78 2022-02-02 02:35:42,302 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.78 2022-02-02 02:35:52,338 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.78 2022-02-02 02:36:02,381 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:36:12,421 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:36:22,457 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:36:32,493 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:36:42,532 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:36:52,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:37:02,609 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:37:12,648 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.83 2022-02-02 02:37:22,684 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 79.89 2022-02-02 02:37:32,726 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:37:42,765 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:37:52,799 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:38:02,838 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:38:12,878 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:38:22,918 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:38:32,961 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:38:42,999 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:38:53,036 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:39:03,072 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:39:13,108 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:39:23,145 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.0 2022-02-02 02:39:33,187 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.05 2022-02-02 02:39:43,223 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.11 2022-02-02 02:39:53,263 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.22 2022-02-02 02:40:03,299 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.22 2022-02-02 02:40:13,338 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.27 2022-02-02 02:40:23,375 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.27 2022-02-02 02:40:33,414 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.43 2022-02-02 02:40:43,456 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.54 2022-02-02 02:40:53,494 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.6 2022-02-02 02:41:03,532 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.65 2022-02-02 02:41:13,568 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.65 2022-02-02 02:41:23,605 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.65 2022-02-02 02:41:33,641 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.65 2022-02-02 02:41:43,674 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.65 2022-02-02 02:41:53,710 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.7 2022-02-02 02:42:03,746 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.76 2022-02-02 02:42:13,780 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.76 2022-02-02 02:42:23,819 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.76 2022-02-02 02:42:33,855 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.81 2022-02-02 02:42:43,891 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.98 2022-02-02 02:42:53,930 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.98 2022-02-02 02:43:03,969 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.98 2022-02-02 02:43:14,012 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 80.98 2022-02-02 02:43:24,058 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.08 2022-02-02 02:43:34,095 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.08 2022-02-02 02:43:44,132 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.08 2022-02-02 02:43:54,173 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.08 2022-02-02 02:44:04,209 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.14 2022-02-02 02:44:14,250 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.3 2022-02-02 02:44:24,290 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.3 2022-02-02 02:44:34,329 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.3 2022-02-02 02:44:44,374 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.35 2022-02-02 02:44:54,411 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.41 2022-02-02 02:45:04,450 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.41 2022-02-02 02:45:14,486 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.46 2022-02-02 02:45:24,525 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.46 2022-02-02 02:45:34,562 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.52 2022-02-02 02:45:44,608 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.52 2022-02-02 02:45:54,648 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.52 2022-02-02 02:46:04,690 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.63 2022-02-02 02:46:14,733 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.84 2022-02-02 02:46:24,773 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 81.95 2022-02-02 02:46:34,815 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.01 2022-02-02 02:46:44,854 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:46:54,892 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:47:04,934 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:47:14,980 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:47:25,020 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:47:35,062 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:47:45,099 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:47:55,138 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.06 2022-02-02 02:48:05,177 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:48:15,214 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:48:25,252 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:48:35,288 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:48:45,331 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:48:55,368 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:49:05,410 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.12 2022-02-02 02:49:15,450 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.17 2022-02-02 02:49:25,489 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.17 2022-02-02 02:49:35,526 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.17 2022-02-02 02:49:45,562 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.17 2022-02-02 02:49:55,598 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.17 2022-02-02 02:50:05,632 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.22 2022-02-02 02:50:15,667 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.22 2022-02-02 02:50:25,703 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.22 2022-02-02 02:50:35,736 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.22 2022-02-02 02:50:45,773 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.22 2022-02-02 02:50:55,808 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.28 2022-02-02 02:51:05,848 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.39 2022-02-02 02:51:15,884 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.39 2022-02-02 02:51:25,917 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.39 2022-02-02 02:51:35,953 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.39 2022-02-02 02:51:45,993 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.39 2022-02-02 02:51:56,029 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.44 2022-02-02 02:52:06,065 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.44 2022-02-02 02:52:16,101 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.44 2022-02-02 02:52:26,138 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.44 2022-02-02 02:52:36,180 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.44 2022-02-02 02:52:46,219 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.44 2022-02-02 02:52:56,255 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.55 2022-02-02 02:53:06,308 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.66 2022-02-02 02:53:16,345 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.71 2022-02-02 02:53:26,380 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.71 2022-02-02 02:53:36,414 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.71 2022-02-02 02:53:46,450 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.71 2022-02-02 02:53:56,489 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.76 2022-02-02 02:54:06,529 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.76 2022-02-02 02:54:16,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.87 2022-02-02 02:54:26,607 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.93 2022-02-02 02:54:36,651 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.98 2022-02-02 02:54:46,694 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 82.98 2022-02-02 02:54:56,737 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:55:06,780 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:55:16,816 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:55:26,867 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:55:36,905 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:55:46,944 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:55:56,982 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.09 2022-02-02 02:56:07,017 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:56:17,055 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:56:27,092 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:56:37,132 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:56:47,170 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:56:57,210 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:57:07,252 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:57:17,289 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.14 2022-02-02 02:57:27,325 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:57:37,375 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:57:47,411 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:57:57,451 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:58:07,490 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:58:17,539 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:58:27,585 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:58:37,625 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:58:47,661 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:58:57,694 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.25 2022-02-02 02:59:07,730 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.31 2022-02-02 02:59:17,765 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.36 2022-02-02 02:59:27,806 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.36 2022-02-02 02:59:37,845 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.36 2022-02-02 02:59:47,884 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.36 2022-02-02 02:59:57,918 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.42 2022-02-02 03:00:07,953 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.42 2022-02-02 03:00:17,992 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.47 2022-02-02 03:00:28,030 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.52 2022-02-02 03:00:38,069 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.58 2022-02-02 03:00:48,117 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.58 2022-02-02 03:00:58,151 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.63 2022-02-02 03:01:08,187 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.63 2022-02-02 03:01:18,226 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.63 2022-02-02 03:01:28,263 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.63 2022-02-02 03:01:38,305 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.69 2022-02-02 03:01:48,341 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.69 2022-02-02 03:01:58,390 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.74 2022-02-02 03:02:08,430 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.74 2022-02-02 03:02:18,466 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.74 2022-02-02 03:02:28,500 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.74 2022-02-02 03:02:38,538 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.74 2022-02-02 03:02:48,572 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.8 2022-02-02 03:02:58,607 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.85 2022-02-02 03:03:08,641 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 83.96 2022-02-02 03:03:18,676 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.12 2022-02-02 03:03:28,717 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.12 2022-02-02 03:03:38,750 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.18 2022-02-02 03:03:48,789 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.23 2022-02-02 03:03:58,822 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.28 2022-02-02 03:04:08,855 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.28 2022-02-02 03:04:18,888 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.28 2022-02-02 03:04:28,923 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.39 2022-02-02 03:04:38,959 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.56 2022-02-02 03:04:48,996 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.56 2022-02-02 03:04:59,032 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.78 2022-02-02 03:05:09,073 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.83 2022-02-02 03:05:19,114 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.83 2022-02-02 03:05:29,154 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.83 2022-02-02 03:05:39,191 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.83 2022-02-02 03:05:49,229 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.88 2022-02-02 03:05:59,266 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.88 2022-02-02 03:06:09,305 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.94 2022-02-02 03:06:19,341 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 84.99 2022-02-02 03:06:29,382 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.05 2022-02-02 03:06:39,424 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.05 2022-02-02 03:06:49,460 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.16 2022-02-02 03:06:59,496 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.27 2022-02-02 03:07:09,532 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.38 2022-02-02 03:07:19,569 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.38 2022-02-02 03:07:29,607 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.49 2022-02-02 03:07:39,647 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.81 2022-02-02 03:07:49,686 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.87 2022-02-02 03:07:59,724 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 85.92 2022-02-02 03:08:09,763 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.03 2022-02-02 03:08:19,802 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.14 2022-02-02 03:08:29,844 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.14 2022-02-02 03:08:39,884 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.25 2022-02-02 03:08:49,927 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.25 2022-02-02 03:08:59,970 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.25 2022-02-02 03:09:10,013 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.31 2022-02-02 03:09:20,046 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.41 2022-02-02 03:09:30,082 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.41 2022-02-02 03:09:40,121 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.41 2022-02-02 03:09:50,161 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.58 2022-02-02 03:10:00,197 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.63 2022-02-02 03:10:10,232 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.69 2022-02-02 03:10:20,269 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.79 2022-02-02 03:10:30,305 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 86.85 2022-02-02 03:10:40,338 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.01 2022-02-02 03:10:50,371 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.23 2022-02-02 03:11:00,407 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.23 2022-02-02 03:11:10,444 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.23 2022-02-02 03:11:20,480 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.23 2022-02-02 03:11:30,513 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.5 2022-02-02 03:11:40,546 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.61 2022-02-02 03:11:50,578 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.77 2022-02-02 03:12:00,617 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.77 2022-02-02 03:12:10,651 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 87.88 2022-02-02 03:12:20,691 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.04 2022-02-02 03:12:30,730 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.04 2022-02-02 03:12:40,763 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.1 2022-02-02 03:12:50,799 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.15 2022-02-02 03:13:00,832 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.15 2022-02-02 03:13:10,868 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.21 2022-02-02 03:13:20,904 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.26 2022-02-02 03:13:30,937 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.26 2022-02-02 03:13:40,976 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.32 2022-02-02 03:13:51,010 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.37 2022-02-02 03:14:01,048 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.42 2022-02-02 03:14:11,082 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.64 2022-02-02 03:14:21,118 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.69 2022-02-02 03:14:31,151 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.8 2022-02-02 03:14:41,186 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 88.8 2022-02-02 03:14:51,219 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 89.08 2022-02-02 03:15:01,250 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 89.18 2022-02-02 03:15:11,286 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 89.4 2022-02-02 03:15:21,318 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 89.67 2022-02-02 03:15:31,358 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 89.78 2022-02-02 03:15:41,392 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.05 2022-02-02 03:15:51,427 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.1 2022-02-02 03:16:01,463 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.21 2022-02-02 03:16:11,497 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.27 2022-02-02 03:16:21,533 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.38 2022-02-02 03:16:31,568 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.48 2022-02-02 03:16:41,599 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.65 2022-02-02 03:16:51,631 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 90.92 2022-02-02 03:17:01,667 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.03 2022-02-02 03:17:11,701 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.19 2022-02-02 03:17:21,734 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.19 2022-02-02 03:17:31,773 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.3 2022-02-02 03:17:41,809 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.52 2022-02-02 03:17:51,842 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.68 2022-02-02 03:18:01,877 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.9 2022-02-02 03:18:11,911 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 91.9 2022-02-02 03:18:21,946 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.11 2022-02-02 03:18:31,977 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.22 2022-02-02 03:18:42,013 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.39 2022-02-02 03:18:52,046 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.44 2022-02-02 03:19:02,082 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.49 2022-02-02 03:19:12,118 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.66 2022-02-02 03:19:22,154 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.82 2022-02-02 03:19:32,190 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.82 2022-02-02 03:19:42,224 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 92.88 2022-02-02 03:19:52,257 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.04 2022-02-02 03:20:02,299 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.31 2022-02-02 03:20:12,338 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.37 2022-02-02 03:20:22,374 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.58 2022-02-02 03:20:32,410 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.64 2022-02-02 03:20:42,447 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.64 2022-02-02 03:20:52,484 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.64 2022-02-02 03:21:02,519 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.69 2022-02-02 03:21:12,558 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.75 2022-02-02 03:21:22,589 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.8 2022-02-02 03:21:32,624 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.8 2022-02-02 03:21:42,664 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.86 2022-02-02 03:21:52,697 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.91 2022-02-02 03:22:02,733 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.91 2022-02-02 03:22:12,779 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 93.97 2022-02-02 03:22:22,819 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.18 2022-02-02 03:22:32,852 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.18 2022-02-02 03:22:42,891 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.18 2022-02-02 03:22:52,921 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.35 2022-02-02 03:23:02,957 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.4 2022-02-02 03:23:12,993 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.46 2022-02-02 03:23:23,028 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.56 2022-02-02 03:23:33,062 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.73 2022-02-02 03:23:43,096 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 94.89 2022-02-02 03:23:53,144 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.06 2022-02-02 03:24:03,180 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.17 2022-02-02 03:24:13,226 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.22 2022-02-02 03:24:23,259 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.33 2022-02-02 03:24:33,296 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.44 2022-02-02 03:24:43,329 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.65 2022-02-02 03:24:53,368 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 95.87 2022-02-02 03:25:03,407 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.09 2022-02-02 03:25:13,444 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.25 2022-02-02 03:25:23,479 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.3 2022-02-02 03:25:33,517 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.3 2022-02-02 03:25:43,552 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.36 2022-02-02 03:25:53,586 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.47 2022-02-02 03:26:03,628 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.47 2022-02-02 03:26:13,664 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.52 2022-02-02 03:26:23,704 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.52 2022-02-02 03:26:33,740 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.52 2022-02-02 03:26:43,775 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.57 2022-02-02 03:26:53,808 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.63 2022-02-02 03:27:03,844 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.74 2022-02-02 03:27:13,881 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.85 2022-02-02 03:27:23,915 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.85 2022-02-02 03:27:33,950 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 96.96 2022-02-02 03:27:43,984 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.07 2022-02-02 03:27:54,022 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.18 2022-02-02 03:28:04,055 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.18 2022-02-02 03:28:14,092 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.18 2022-02-02 03:28:24,140 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.18 2022-02-02 03:28:34,217 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.18 2022-02-02 03:28:44,250 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.34 2022-02-02 03:28:54,285 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.34 2022-02-02 03:29:04,319 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.39 2022-02-02 03:29:14,354 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.45 2022-02-02 03:29:24,391 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.61 2022-02-02 03:29:34,427 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.67 2022-02-02 03:29:44,463 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.72 2022-02-02 03:29:54,500 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.78 2022-02-02 03:30:04,532 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.88 2022-02-02 03:30:14,565 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.88 2022-02-02 03:30:24,601 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 97.99 2022-02-02 03:30:34,637 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.16 2022-02-02 03:30:44,671 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.26 2022-02-02 03:30:54,704 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.26 2022-02-02 03:31:04,736 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.26 2022-02-02 03:31:14,773 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.26 2022-02-02 03:31:24,805 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.37 2022-02-02 03:31:34,838 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.37 2022-02-02 03:31:44,874 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.48 2022-02-02 03:31:54,907 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.54 2022-02-02 03:32:04,940 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.54 2022-02-02 03:32:14,973 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.65 2022-02-02 03:32:25,009 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.75 2022-02-02 03:32:35,043 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.75 2022-02-02 03:32:45,082 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.75 2022-02-02 03:32:55,117 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.92 2022-02-02 03:33:05,148 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.92 2022-02-02 03:33:15,178 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 98.97 2022-02-02 03:33:25,210 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.03 2022-02-02 03:33:35,247 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.03 2022-02-02 03:33:45,309 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.08 2022-02-02 03:33:55,345 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.08 2022-02-02 03:34:05,384 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.13 2022-02-02 03:34:15,424 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.13 2022-02-02 03:34:25,463 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.24 2022-02-02 03:34:35,497 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.24 2022-02-02 03:34:45,540 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.24 2022-02-02 03:34:55,576 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.3 2022-02-02 03:35:05,611 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.46 2022-02-02 03:35:15,651 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.46 2022-02-02 03:35:25,694 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.46 2022-02-02 03:35:35,733 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.46 2022-02-02 03:35:45,766 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.51 2022-02-02 03:35:55,801 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 03:36:05,836 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 03:36:15,875 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:36:25,911 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:36:35,948 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:36:45,982 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:36:56,013 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:37:06,056 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:37:16,096 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:37:26,128 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:37:36,161 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:37:46,194 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:37:56,234 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.62 2022-02-02 03:38:06,276 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.68 2022-02-02 03:38:16,312 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.73 2022-02-02 03:38:26,345 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.73 2022-02-02 03:38:36,381 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.73 2022-02-02 03:38:46,420 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.73 2022-02-02 03:38:56,457 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 03:39:06,499 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:39:16,536 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:39:26,575 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:39:36,611 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:39:46,660 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:39:56,697 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:40:06,730 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:40:16,766 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:40:26,802 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:40:36,838 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:40:46,875 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 03:40:56,911 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.89 2022-02-02 03:41:06,950 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.95 2022-02-02 03:41:16,984 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.95 2022-02-02 03:41:27,019 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.95 2022-02-02 03:41:37,055 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: running, progress: 99.95 2022-02-02 03:41:52,107 | test | INFO | pool-3-thread-29 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 03:41:52,154 | test | INFO | pool-3-thread-29 | [task:check:468] Rebalance completed with progress: 100% in 8259.76799989 sec 2022-02-02 03:54:14,565 | test | INFO | pool-3-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****..........................***.........*****X | 4233.300 | | 1961.700 | ...............*X | 1961.700 | | 1210.600 | .........*****X | 1210.600 | | 495.600 | ...*X | 495.600 | | 285.200 | .**************..**X | 502.300 | | 494.200 | ...****X | 494.200 | | 333.900 | .*********...**.*****X | 762.400 | | 709.000 | ....**.*****X | 762.700 | | 753.700 | .....*X | 753.700 | | 660.300 | ....*****X | 660.300 | | 624.200 | ....********.**X | 833.900 | | 820.900 | .....******.********.**.*****X | 1151.200 | | 913.600 | ......*X | 913.600 | | 748.500 | .....*****X | 748.500 | | 713.700 | ....**.******.*********.*X | 991.500 | | 972.600 | .......*******X | 974.200 | | 951.900 | ......*****X | 951.900 | | 848.200 | ......*X | 848.200 | | 844.900 | ......*****X | 844.900 | | 727.800 | .....*X | 727.800 | | 639.600 | ....******.***.*****.***.******X | 1118.400 | | 1091.200 | ........X | 1091.200 | | 1023.800 | .......*****X | 1023.800 | | 992.800 | .......*X | 992.800 | | 919.800 | ......*****X | 919.800 | | 731.700 | .....*******X | 751.600 | | 650.700 | ....**X | 650.700 | | 560.500 | ...******.**.******..*X | 1025.300 | | 980.000 | .......*****X | 980.000 | | 693.200 | ....**X | 693.200 | | 641.400 | ....*****.***.*****X | 892.400 | | 824.600 | .....*X | 824.600 | | 714.700 | ....*****X | 714.700 | | 634.400 | ....*******X | 697.700 | | 671.000 | ....X | 671.000 | | 602.600 | ....X | 602.600 | | 544.200 | ...******.**.******.*X | 912.700 | | 801.700 | .....******.**.******...**.*****X | 1455.800 | | 1280.100 | .........**X | 1280.100 | | 1171.200 | ........*****X | 1171.200 | | 872.600 | ......********...*******X | 1309.700 | | 1202.000 | ........**X | 1202.000 | | 1092.100 | ........****X | 1092.100 | | 1001.600 | .......*********.....**.*****X | 1796.400 | | 1519.500 | ...........*X | 1519.500 | | 1344.900 | ..........*********.*****X | 1510.200 | | 1441.100 | ..........*X | 1441.100 | | 1335.600 | ..........*****X | 1335.600 | | 983.300 | .......*X | 983.300 | | 964.800 | .......******..********.***.****X | 1486.900 | | 1349.500 | ..........**X | 1349.500 | | 1292.400 | .........*****..**X | 1534.100 | | 1523.000 | ...........****X | 1523.000 | | 1261.100 | .........X | 1261.100 | | 1126.100 | ........********....*X | 1564.900 | | 1499.000 | ...........*****X | 1499.000 | | 1084.700 | ........*X | 1084.700 | | 1016.400 | .......*****X | 1016.400 | | 904.600 | ......*X | 904.600 | | 835.100 | .....*****X | 835.100 | | 563.500 | ...*X | 563.500 | | 368.700 | ..*****X | 368.700 | | 320.600 | .**X | 320.600 | | 317.800 | .*****.**.*.********.*X | 798.300 | | 798.300 | .....****.*X | 898.500 | | 708.200 | ....*****X | 708.200 | | 677.700 | ....***.****X | 775.700 | | 465.800 | ..**X | 465.800 | | 398.300 | ..*****.********X | 590.500 | | 519.800 | ...**.******....*X | 1202.300 | | 1086.100 | ........******.**..******..**.*****X | 1920.600 | | 1732.600 | .............**X | 1732.600 | | 1367.500 | ..........******..**....*****X | 2096.000 | | 2084.900 | ................*X | 2084.900 | | 1874.300 | ..............*****X | 1874.300 | | 1203.800 | .........**X | 1203.800 | | 943.700 | ......****X | 943.700 | | 669.400 | ....*******.*..**X | 1039.900 | | 877.200 | ......********..******...**...******................*****.**X | 3901.300 | | 1834.900 | ..............**X | 1834.900 | | 1499.200 | ...........*****.....****.***X | 2278.100 | | 1846.800 | ..............**X | 1846.800 | | 1676.100 | ............******....*******X | 2116.400 | | 1518.100 | ...........**.******...........**.*****X | 3035.000 | | 2709.500 | .....................**..******.**X | 3078.100 | | 2699.300 | .....................****X | 2699.300 | | 2177.600 | .................***...*****..............*.......**....*****X | 5570.300 | | 2946.500 | .......................*X | 2946.500 | | 2262.800 | .................******..........**....******..**X | 4139.100 | | 3973.700 | ................................*****X | 3973.700 | | 3768.300 | ..............................**...******............**.....******.........*X | 7266.300 | | 6591.900 | .....................................................*****X | 6591.900 | | 3219.500 | .........................*X | 3219.500 | | 2972.100 | .......................******............**X | 4369.600 | | 4353.700 | ...................................****X | 4353.700 | | 2017.200 | ...............**X | 2017.200 | | 1543.400 | ...........******......**..******...............**........******.....................**....*****X | 8204.300 | | 5675.700 | ..............................................*X | 5675.700 | | 4861.400 | .......................................*****X | 4861.400 | | 4610.200 | .....................................***.*****..............................***......*****X | 9071.100 | | 5459.900 | ............................................*X | 5459.900 | | 4657.900 | .....................................******........**.*****X | 5729.400 | | 3828.800 | ..............................*X | 3828.800 | | 3365.900 | ..........................*****X | 3365.900 | | 2983.700 | .......................**X | 2983.700 | | 2890.300 | .......................*****........********X | 3917.300 | | 3417.000 | ...........................********............**...******...........**........******....................***.****X | 10035.400 | | 5640.500 | .............................................**X | 5640.500 | | 3976.500 | ................................*****X | 3976.500 | | 2482.800 | ...................*X | 2482.800 | | 2237.800 | .................******..........**.......*X | 4288.100 | | 4288.100 | ..................................***X | 4288.100 | | 4045.600 | ................................*X | 4045.600 | | 3645.200 | .............................*****X | 3645.200 | | 3307.400 | ..........................**X | 3307.400 | | 3231.800 | .........................*****.*.**.******.......................**.....*****X | 6910.900 | | 4680.900 | .....................................*X | 4680.900 | | 4533.800 | ....................................******.......**X | 5361.400 | | 5031.500 | ........................................*****...***.....X | 5977.800 | | 5977.800 | ................................................****........................................***...........*****X | 12024.800 | | 7918.700 | ................................................................*X | 7918.700 | | 6258.600 | ...................................................*****X | 6258.600 | | 5376.200 | ...........................................**...X | 5673.500 | | 5673.500 | ..............................................***X | 5673.500 | | 5673.500 | ..............................................X | 5673.500 | | 5500.600 | ............................................*X | 5500.600 | | 5333.500 | ...........................................*****X | 5333.500 | | 4602.600 | .....................................*X | 4602.600 | | 3925.100 | ...............................*****X | 3925.100 | | 1566.600 | ............*X | 1566.600 | | 1117.000 | ........******........***...*****X | 2494.900 | | 2150.000 | ................*X | 2150.000 | | 1984.300 | ...............*****X | 1984.300 | | 1138.000 | ........*X | 1138.000 | | 920.000 | ......******.....***..*****................................................X | 7503.800 | | 7503.800 | .............................................................**................*****X | 9414.000 | | 3023.200 | ........................*X | 3023.200 | | 929.100 | ......******........**..******......................***........****X | 5757.900 | | 4904.600 | .......................................**X | 4904.600 | | 4282.300 | ..................................*****X | 4282.300 | | 2402.200 | ..................X | 2402.200 | | 1404.200 | ..........X | 1404.200 | | 701.800 | ....******.....**..*****X | 1486.900 | | 1470.100 | ...........**X | 1470.100 | | 1319.200 | .........****X | 1319.200 | | 748.800 | .....**X | 748.800 | | 645.000 | ....******...**..******....*******X | 1779.700 | | 1571.300 | ............*X | 1571.300 | | 1560.800 | ...........*****X | 1560.800 | | 1302.800 | .........**X | 1302.800 | | 1098.400 | ........****X | 1098.400 | | 761.000 | .....***.******.....*X | 1462.200 | | 1387.100 | ..........*****X | 1387.100 | | 885.100 | ......*X | 885.100 | | 880.900 | ......*****X | 880.900 | | 780.700 | .....******..**......**X | 1710.500 | | 1427.200 | ..........****X | 1427.200 | | 1408.400 | ..........***..*****X | 1615.900 | | 1087.200 | ........*X | 1087.200 | | 934.000 | ......****X | 934.000 | | 934.000 | ......*...*******X | 1263.700 | | 1014.200 | .......*X | 1014.200 | | 975.500 | .......******....***.****X | 1589.000 | | 1110.500 | ........**X | 1110.500 | | 950.900 | ......****X | 950.900 | | 789.100 | .....*X | 789.100 | | 789.100 | .....*.**X | 843.000 | | 843.000 | ......***...**.******....**.*****X | 2015.000 | | 1778.100 | .............*X | 1778.100 | | 1643.400 | ............*****X | 1643.400 | | 1280.700 | .........**X | 1280.700 | | 1188.300 | ........******.**.**************..**.*****X | 1693.400 | | 1596.000 | ............***.*****X | 1710.500 | | 775.800 | .....*X | 775.800 | | 714.200 | ....*****X | 714.200 | | 456.000 | ..*X | 456.000 | | 386.900 | ..*****X | 386.900 | | 386.700 | ..*X | 386.700 | | 380.200 | ..****************X | 384.800 | | 380.100 | ..***************X | 435.100 | | 375.500 | ..***X | 375.500 | | 375.500 | ..**.........***..****X | 1760.000 | | 695.300 | ....**X | 695.300 | | 478.300 | ..******.**.*****X | 697.200 | | 518.100 | ...*X | 518.100 | | 401.000 | ..**X | 401.000 | | 401.000 | ..****X | 460.600 | | 460.600 | ..X | 460.600 | | 322.900 | .********..*****X | 484.800 | | 425.100 | ..*X | 425.100 | | 404.500 | ..******.*********...**..*****X | 1097.100 | | 660.500 | ....*X | 660.500 | | 256.200 | .******....**........******.......................***..****X | 4775.600 | | 3741.900 | ..............................X | 3741.900 | | 1752.600 | .............*X | 1752.600 | | 897.100 | ......******..**....*****X | 1628.100 | | 1514.100 | ...........*X | 1514.100 | | 1453.600 | ...........*****X | 1453.600 | | 1135.700 | ........**X | 1135.700 | | 904.300 | ......*****................***......*****X | 3575.800 | | 2919.300 | .......................*X | 2919.300 | | 2806.700 | ......................*****X | 2806.700 | | 2612.500 | ....................*X | 2612.500 | | 2349.000 | ..................*****X | 2349.000 | | 1469.200 | ...........**X | 1469.200 | | 944.500 | ......*****............***......*****X | 3057.900 | | 1827.600 | ..............*X | 1827.600 | | 1248.600 | .........*****X | 1248.600 | | 995.400 | .......*X | 995.400 | | 588.300 | ...******.***..***X | 898.500 | | 898.500 | ......*...***..******......**...*****X | 2592.400 | | 1727.900 | .............*X | 1727.900 | | 1343.100 | ..........******...**.*****X | 1876.200 | | 1610.500 | ............**X | 1610.500 | | 1206.600 | .........*****X | 1206.600 | | 695.800 | ....*X | 695.800 | | 561.400 | ...*****X | 561.400 | | 471.200 | ..*X | 471.200 | | 412.000 | ..******..**....******...*******X | 1563.100 | | 1255.200 | .........**X | 1255.200 | | 1120.300 | ........******...................**............***X | 4816.000 | | 4816.000 | .......................................*X | 4816.000 | | 2800.100 | ......................*X | 2800.100 | | 2060.700 | ................******.....*X | 2656.200 | | 1698.800 | .............*****X | 1698.800 | | 1172.600 | ........**X | 1172.600 | | 798.700 | .....******.*******X | 936.500 | | 587.800 | ...*X | 587.800 | | 541.100 | ...*****X | 541.100 | | 530.700 | ...X | 530.700 | | 512.900 | ...*X | 512.900 | | 508.800 | ...****X | 508.800 | | 387.700 | ..**X | 387.700 | | 382.100 | ..**************.********...***.*******X | 1029.200 | | 1009.800 | .......*******X | 1047.800 | | 1018.100 | .......*****X | 1018.100 | | 790.600 | .....*X | 790.600 | | 754.300 | .....*****X | 754.300 | | 740.400 | .....****************..**X | 1022.700 | | 979.900 | .......*****X | 979.900 | | 713.700 | ....**.******....**.******..*X | 1617.800 | | 1613.100 | ............*****X | 1613.100 | | 969.200 | .......*X | 969.200 | | 969.200 | .......X | 969.200 | | 880.200 | ......*X | 880.200 | | 880.200 | ......**X | 880.200 | | 461.900 | ..**X | 461.900 | | 438.800 | ..******...***************X | 838.500 | | 671.800 | ....**X | 671.800 | | 662.000 | ....*****.**X | 781.500 | | 764.100 | .....*****X | 764.100 | | 403.800 | ..*X | 403.800 | | 372.900 | ..*****X | 372.900 | | 299.500 | .*X | 299.500 | | 290.700 | .*************X | 306.000 | | 295.600 | .************************X | 305.800 | | 290.900 | .****************............***...*****X | 2113.300 | | 846.000 | ......*X | 846.000 | | 431.500 | ..*****X | 431.500 | | 316.100 | .*******X | 316.100 | | 270.500 | .*********.*******X | 388.200 | | 319.900 | .*X | 319.900 | | 314.800 | .******...*******X | 696.000 | | 684.700 | ....*********.*X | 806.700 | | 794.200 | .....*****X | 794.200 | | 585.100 | ...*X | 585.100 | | 569.300 | ...*****X | 569.300 | | 266.100 | .*X | 266.100 | | 233.200 | ***************.********...*X | 634.300 | | 618.300 | ....*******X | 635.900 | | 618.700 | ....*****X | 618.700 | | 388.800 | ..*****************.*X | 581.100 | | 546.400 | ...******..*******X | 765.800 | | 495.200 | ...*X | 495.200 | | 490.600 | ...******.**.*****X | 726.200 | | 590.500 | ...*X | 590.500 | | 590.400 | ...******.*X | 619.600 | | 613.400 | ....******...*******X | 1052.800 | | 487.500 | ...**X | 487.500 | | 470.400 | ..******....*X | 947.200 | | 946.700 | ......*****X | 946.700 | | 658.000 | ....********..*******X | 943.100 | | 774.700 | .....*X | 774.700 | | 774.700 | .....X | 774.700 | | 693.700 | ....***X | 693.700 | | 693.700 | ....**..*******X | 956.600 | | 828.900 | .....*X | 828.900 | | 821.800 | .....*****X | 821.800 | | 806.200 | .....********......***...******.....*....X | 2891.400 | | 2496.900 | ...................*****X | 2496.900 | | 746.100 | .....*X | 746.100 | | 578.300 | ...*****X | 578.300 | | 466.700 | ..*******X | 480.200 | | 384.400 | ..*X | 384.400 | | 376.500 | ..*****X | 376.500 | | 260.500 | .**X | 260.500 | | 233.800 | *****...*****X | 523.200 | | 523.200 | ...**X | 523.200 | | 410.900 | ..*X | 410.900 | | 378.000 | ..******.****************X | 582.600 | | 509.300 | ...********...*X | 907.100 | | 871.600 | ......*****X | 871.600 | | 645.500 | ....********.....**X | 1300.100 | | 1293.500 | .........*****X | 1293.500 | | 609.400 | ....*X | 609.400 | | 577.200 | ...******.**.******.**X | 896.800 | | 859.800 | ......*******X | 926.800 | | 911.400 | ......*X | 911.400 | | 911.400 | ......***X | 911.400 | | 797.200 | .....********.....*X | 1334.400 | | 1306.300 | .........*****X | 1306.300 | | 1112.400 | ........*******X | 1154.500 | | 723.900 | .....**X | 723.900 | | 693.200 | ....******...*X | 1029.300 | | 1014.500 | .......********.*****X | 1094.200 | | 986.100 | .......**X | 986.100 | | 952.700 | ......*****....**X | 1362.700 | | 1355.000 | ..........*****X | 1355.000 | | 986.200 | .......**********.********X | 1135.000 | | 1031.800 | .......****X | 1031.800 | | 651.400 | ....**X | 651.400 | | 600.900 | ...*****X | 600.900 | | 557.000 | ...*X | 557.000 | | 527.100 | ...*************X | 590.700 | | 413.700 | ..**X | 413.700 | | 383.900 | ..*****....*********..*X | 1088.600 | | 1084.400 | ........*****X | 1084.400 | | 859.000 | ......*X | 859.000 | | 849.400 | ......*****X | 849.400 | | 748.900 | .....*********...********.*X | 1302.900 | | 1210.800 | .........*****X | 1210.800 | | 969.100 | .......X | 969.100 | | 969.100 | .......******X | 995.300 | | 718.700 | ....**X | 718.700 | | 687.400 | ....******.********.*X | 921.500 | | 921.500 | ......X | 921.500 | | 892.200 | ......*X | 892.200 | | 892.200 | ......**X | 892.200 | | 700.900 | ....*********..*******X | 861.600 | | 734.400 | .....*X | 734.400 | | 722.400 | .....******....*******X | 1208.300 | | 1044.800 | .......***.*****X | 1104.400 | | 940.100 | ......*X | 940.100 | | 904.000 | ......*****X | 904.000 | | 849.300 | ......********......*******X | 1676.800 | | 1445.900 | ...........**X | 1445.900 | | 1294.400 | .........*****X | 1294.400 | | 1174.500 | ........**.*****X | 1241.000 | | 646.800 | ....*X | 646.800 | | 551.100 | ...*****X | 551.100 | | 465.300 | ..*X | 465.300 | | 448.200 | ..*****X | 448.200 | | 438.500 | ..X | 438.500 | | 433.700 | ..*******X | 478.400 | | 465.600 | ..*X | 465.600 | | 422.400 | ..*****X | 422.400 | | 228.200 | *X | 228.200 | | 221.800 | *****X | 221.800 | | 220.600 | *********.********.**.******..*X | 758.800 | | 710.700 | ....*****X | 710.700 | | 596.600 | ...***.****X | 631.400 | | 308.500 | .**X | 308.500 | | 279.200 | .******..*********X | 568.200 | | 557.600 | ...******.*******X | 646.500 | | 595.900 | ...**.********X | 649.800 | | 592.300 | ...****X | 592.300 | | 547.400 | ...***.*******X | 634.500 | | 611.900 | ....******.*********X | 841.700 | | 835.100 | .....*****X | 835.100 | | 689.500 | ....**X | 689.500 | | 659.700 | ....*****X | 659.700 | | 554.000 | ...*X | 554.000 | | 549.800 | ...******..**.*****X | 858.500 | | 700.400 | ....**X | 700.400 | | 652.400 | ....****X | 652.400 | | 626.200 | ....**********X | 654.400 | | 649.500 | ....*****X | 649.500 | | 624.200 | ....*******X | 624.300 | | 607.500 | ....********.*********..*X | 1005.300 | | 996.700 | .......*****X | 996.700 | | 759.600 | .....*X | 759.600 | | 714.400 | ....*****X | 714.400 | | 710.400 | ....***.*******X | 796.200 | | 786.600 | .....*****X | 786.600 | | 631.600 | ....*X | 631.600 | | 627.500 | ....*******X | 652.800 | | 651.800 | ....*****X | 651.800 | | 494.300 | ...*X | 494.300 | | 479.000 | ..******....*******X | 938.200 | | 755.900 | .....**X | 755.900 | | 712.500 | ....*****.*********.*X | 886.300 | | 808.200 | .....******..***.*****.**....******.........**X | 2773.200 | | 2258.300 | .................*****X | 2258.300 | | 938.800 | ......*X | 938.800 | | 929.600 | ......*******X | 930.500 | | 904.800 | ......******..*******X | 1161.400 | | 716.800 | ....**X | 716.800 | | 709.600 | ....******...*X | 1072.200 | | 1069.500 | .......*****X | 1069.500 | | 680.900 | ....*X | 680.900 | | 657.500 | ....******.********....**********X | 1294.400 | | 1274.800 | .........*****X | 1274.800 | | 694.600 | ....*X | 694.600 | | 683.600 | ....******....**X | 1182.000 | | 1181.000 | ........****X | 1181.000 | | 1121.100 | ........********X | 1152.100 | | 819.400 | .....********...*******X | 1164.100 | | 1022.900 | .......**X | 1022.900 | | 975.700 | .......******.*******X | 1198.000 | | 995.600 | .......*******X | 1036.500 | | 936.900 | ......*X | 936.900 | | 860.900 | ......******.**X | 1064.900 | | 1058.600 | .......*****X | 1058.600 | | 680.800 | ....*******X | 686.600 | | 528.800 | ...********...**X | 866.800 | | 853.100 | ......********...******................*******X | 3184.100 | | 834.000 | .....*X | 834.000 | | 410.500 | ..******.**.*****X | 620.300 | | 436.000 | ..**X | 436.000 | | 381.900 | ..*************X | 402.000 | | 381.900 | ..********..*X | 604.300 | | 594.000 | ...*****X | 594.000 | | 328.800 | .**X | 328.800 | | 319.500 | .*****X | 319.500 | | 312.300 | .***************X | 324.000 | | 307.800 | .*****************.********..*******X | 656.200 | | 613.500 | ....**X | 613.500 | | 597.100 | ...*****X | 597.100 | | 504.100 | ...********.*X | 657.800 | | 615.700 | ....******..*******X | 902.700 | | 655.700 | ....**X | 655.700 | | 617.900 | ....******.*******X | 752.800 | | 451.900 | ..**.******...***.****X | 999.800 | | 903.500 | ......**X | 903.500 | | 897.900 | ......******.*******X | 1001.900 | | 691.800 | ....*X | 691.800 | | 632.000 | ....******...*X | 1045.600 | | 1028.500 | .......*****X | 1028.500 | | 643.100 | ....*********..*X | 903.600 | | 890.300 | ......*****X | 890.300 | | 679.100 | ....********..*X | 899.900 | | 892.500 | ......*****X | 892.500 | | 851.300 | ......**X | 851.300 | | 840.100 | .....******..*******X | 1081.700 | | 676.900 | ....*X | 676.900 | | 611.800 | ....******..*X | 845.500 | | 837.200 | .....*X | 837.200 | | 837.200 | .....***X | 837.200 | | 417.900 | ..**X | 417.900 | | 378.100 | ..*****X | 378.100 | | 298.500 | .*******X | 303.200 | | 232.700 | *X | 232.700 | | 228.000 | **************X | 240.000 | | 232.800 | **.******...*X | 669.000 | | 664.100 | ....*****X | 664.100 | | 400.300 | ..**X | 400.300 | | 378.700 | ..******..***X | 670.100 | | 670.100 | ....**X | 670.100 | | 670.100 | ....X | 670.100 | | 549.700 | ...*X | 549.700 | | 535.600 | ...******.**X | 622.500 | | 618.700 | ....*****.***.******....*******X | 1404.200 | | 1042.600 | .......********......***..*****X | 1968.900 | | 1933.600 | ...............*X | 1933.600 | | 1671.300 | ............*****X | 1671.300 | | 343.000 | .*X | 343.000 | | 314.400 | .******.......***.******...*X | 1620.800 | | 1567.400 | ............*****X | 1567.400 | | 645.500 | ....*X | 645.500 | | 590.500 | ...******....***.******............**.*****X | 2656.400 | | 2553.800 | ....................**..*****X | 2844.500 | | 2067.400 | ................*X | 2067.400 | | 1639.800 | ............*****X | 1639.800 | | 1368.200 | ..........**X | 1368.200 | | 1362.700 | ..........*****X | 1362.700 | | 1076.300 | .......*X | 1076.300 | | 1019.900 | .......**************.....***.****X | 1688.000 | | 1134.300 | ........*********.........**.*****X | 2339.100 | | 2248.200 | .................*X | 2248.200 | | 2187.800 | .................*****X | 2187.800 | | 1783.600 | .............***.*****X | 1900.100 | | 1849.600 | ..............*X | 1849.600 | | 1750.100 | .............*****X | 1750.100 | | 1477.100 | ...........*X | 1477.100 | | 1286.100 | .........******..***...******........**..*****X | 3014.100 | | 2308.000 | ..................*X | 2308.000 | | 2246.200 | .................*****X | 2246.200 | | 1741.300 | .............**X | 1741.300 | | 1438.200 | ..........****X | 1438.200 | | 1138.000 | ........**X | 1138.000 | | 1132.700 | ........******.....**...*******X | 2131.800 | | 2045.100 | ................******........***......*****.**X | 3947.700 | | 3512.100 | ............................*****X | 3512.100 | | 2966.000 | .......................*X | 2966.000 | | 2880.400 | ......................*****X | 2880.400 | | 1575.300 | ............**X | 1575.300 | | 1235.600 | .........****X | 1235.600 | | 387.600 | ..**X | 387.600 | | 147.200 | *****X | 147.200 | | 130.500 | *X | 130.500 | | 98.700 | ******..**.*****X | 468.300 | | 341.400 | .**X | 341.400 | | 311.800 | .******.*******X | 455.500 | | 362.800 | ..*******X | 433.600 | | 417.800 | ..***.****X | 509.700 | | 405.500 | ..*********.*X | 505.700 | | 471.000 | ..******...*X | 802.500 | | 774.600 | .....*****X | 774.600 | | 518.100 | ...***..******....*X | 1241.400 | | 1170.000 | ........*****X | 1170.000 | | 1024.000 | .......*X | 1024.000 | | 882.100 | ......*****X | 882.100 | | 841.800 | ......*X | 841.800 | | 752.800 | .....*****X | 752.800 | | 596.500 | ...***.*****X | 687.800 | | 553.400 | ...*X | 553.400 | | 478.200 | ..******...*X | 740.800 | | 711.400 | ....*****X | 711.400 | | 420.400 | ..**X | 420.400 | | 361.400 | ..*************X | 428.600 | | 193.000 | *X | 193.000 | | 117.600 | ******....**.....*****X | 1105.800 | | 646.300 | ....**X | 646.300 | | 278.700 | .*******X | 278.700 | | 246.900 | .******.**.*****X | 541.100 | | 468.700 | ..**X | 468.700 | | 341.500 | .*****.*********.**.*****X | 648.600 | | 516.200 | ...*X | 516.200 | | 444.200 | ..******..**X | 706.900 | | 605.100 | ....******......**....*******X | 1854.100 | | 1663.500 | ............*****X | 1663.500 | | 441.400 | ..*X | 441.400 | | 415.200 | ..******...***.*****X | 843.000 | | 591.800 | ...*X | 591.800 | | 342.100 | .*******X | 343.100 | | 256.900 | .**************.***.*****X | 568.000 | | 373.300 | ..*X | 373.300 | | 254.300 | .******...**.*****X | 800.400 | | 612.600 | ....**X | 612.600 | | 459.400 | ..******.**..******..........**...******.......***..*****X | 3408.600 | | 2043.600 | ...............*X | 2043.600 | | 937.700 | ......*****X | 937.700 | | 102.000 | **.*****X | 199.700 | | 167.700 | **X | 167.700 | | 148.700 | ********.******.**.*****X | 582.300 | | 578.500 | ...**.*****X | 688.900 | | 566.300 | ...**X | 566.300 | | 381.600 | ..******..**.******.*X | 896.100 | | 785.300 | .....*********...******.....*X | 1723.500 | | 1626.500 | ............*****X | 1626.500 | | 617.400 | ....*X | 617.400 | | 294.600 | .******...***..****X | 908.000 | | 571.500 | ...********X | 588.900 | | 349.000 | .*X | 349.000 | | 178.700 | ******..*******X | 480.100 | | 472.400 | ..***.*****X | 545.800 | | 369.100 | ..*X | 369.100 | | 342.700 | .******.*X | 456.400 | | 274.000 | .*****X | 274.000 | | 134.000 | **X | 134.000 | | 127.300 | **********************...*******X | 575.400 | | 291.400 | .**X | 291.400 | | 170.700 | *************X | 234.800 | | 176.000 | *X | 176.000 | | 80.100 | *****X | 80.100 | | 79.300 | *X | 79.300 | | 76.700 | ****************X | 80.000 | | 76.600 | ********.********X | 229.600 | | 193.900 | *****X | 193.900 | | 162.500 | *X | 162.500 | | 91.400 | *****X | 91.400 | | 31.000 | **.******..**.*****X | 510.400 | | 387.100 | ..**X | 387.100 | | 234.900 | ****X | 234.900 | | 139.800 | ****.*****.*X | 449.500 | | 378.300 | ..*****X | 378.300 | | 143.500 | *X | 143.500 | | 72.000 | *********.******.**.*****X | 400.100 | | 364.100 | ..*******X | 462.800 | | 461.000 | ..*X | 461.000 | | 333.600 | .*****X | 333.600 | | 281.400 | .**X | 281.400 | | 168.600 | *****X | 168.600 | | 48.500 | *X | 48.500 | | 13.100 | ******.*******X | 155.500 | | 36.900 | **X | 36.900 | | 5.700 | ****************.*****X | 161.400 | | 71.100 | *X | 71.100 | | 30.000 | **************.***.*****X | 274.900 | | 250.600 | .**.*****X | 468.500 | | 308.600 | .*X | 308.600 | | 169.100 | *****************..******.*******X | 528.900 | | 378.000 | ..**X | 378.000 | | 359.100 | .********.********.*****X | 541.500 | | 540.000 | ...**X | 540.000 | | 395.900 | ..****X | 395.900 | | 337.000 | .**X | 337.000 | | 164.000 | *******X | 164.000 | | 156.100 | *******X | 201.500 | | 168.200 | ******.***.*****X | 369.200 | | 219.500 | *X | 219.500 | | 182.100 | ********.****************X | 347.100 | | 344.200 | .*****X | 344.200 | | 328.200 | .*X | 328.200 | | 220.000 | ******.....**.*****X | 880.100 | | 402.900 | ..**X | 402.900 | | 326.700 | .******.*X | 367.800 | | 338.400 | .******....**..*****X | 1080.000 | | 657.100 | ....**X | 657.100 | | 427.900 | ..******.*X | 539.500 | | 511.700 | ...*****X | 511.700 | | 414.400 | ..*X | 414.400 | | 318.500 | .******.......***....****X | 1606.200 | | 1027.200 | .......**X | 1027.200 | | 731.700 | .....*****X | 731.700 | | 473.400 | ..**..******...********.......***..*****X | 2085.400 | | 981.600 | .......*X | 981.600 | | 684.500 | ....******.......***.....*****...***..*****X | 2675.100 | | 2542.100 | ....................*X | 2542.100 | | 2251.200 | .................*****X | 2251.200 | | 1448.700 | ...........*X | 1448.700 | | 1146.300 | ........******....********X | 1653.000 | | 1309.300 | .........*X | 1309.300 | | 1134.800 | ........*****X | 1134.800 | | 919.500 | ......**.******....***.*****X | 1581.500 | | 1026.500 | .......*X | 1026.500 | | 719.200 | ....******.**.******.**X | 979.500 | | 928.400 | ......****X | 928.400 | | 592.500 | ...**X | 592.500 | | 579.400 | ...*****X | 579.400 | | 575.200 | ...**.******.**X | 762.400 | | 742.300 | .....****X | 742.300 | | 721.900 | .....**X | 721.900 | | 664.700 | ....*****X | 664.700 | | 406.000 | ..*X | 406.000 | | 358.200 | .******...*X | 654.400 | | 560.100 | ...*****X | 560.100 | | 273.200 | .*********.......**..*****X | 1426.000 | | 866.400 | ......*X | 866.400 | | 458.300 | ..*****X | 458.300 | | 215.800 | ***.*****X | 260.700 | | 229.500 | **.******....................**.......******........*X | 4472.800 | | 4252.800 | ..................................******...***..******....**...*****X | 5701.000 | | 5026.100 | ........................................*X | 5026.100 | | 4450.600 | ....................................*****X | 4450.600 | | 2332.900 | ..................**X | 2332.900 | | 2052.700 | ................*****.........................***.....****X | 5709.500 | | 4260.400 | ..................................**X | 4260.400 | | 3728.200 | ..............................*****X | 3728.200 | | 3525.200 | ............................**...*****X | 3891.100 | | 3743.200 | ..............................*X | 3743.200 | | 3425.000 | ...........................******..**..******...**X | 4233.200 | | 4040.800 | ................................*****X | 4040.800 | | 3154.300 | .........................********..*X | 3432.100 | | 3322.900 | ..........................*****X | 3322.900 | | 1976.800 | ...............*X | 1976.800 | | 1417.000 | ..........******..........***...*****X | 2899.500 | | 1300.700 | .........*X | 1300.700 | | 931.200 | ......*****X | 931.200 | | 823.500 | .....**.******..********.**X | 1262.700 | | 1218.200 | .........******.**.******...*X | 1841.300 | | 1816.000 | ..............*****X | 1816.000 | | 1715.600 | .............***.*****X | 1811.400 | | 1260.700 | .........*X | 1260.700 | | 1148.700 | ........******.....*******X | 1709.000 | | 1440.500 | ..........**X | 1440.500 | | 1342.800 | ..........*****X | 1342.800 | | 792.300 | .....*X | 792.300 | | 724.000 | .....******...**.******..........***.*****.........***.....*****X | 4313.500 | | 3351.000 | ..........................*X | 3351.000 | | 2556.700 | ....................*****X | 2556.700 | | 1694.400 | .............*X | 1694.400 | | 1689.600 | .............*****X | 1689.600 | | 1575.500 | ............**X | 1575.500 | | 1473.700 | ...........******...**..******.*X | 2269.300 | | 2164.700 | .................*********.******...*X | 2712.400 | | 2591.400 | ....................*****X | 2591.400 | | 1704.800 | .............********...***.******..*X | 2422.100 | | 2266.300 | .................*****X | 2266.300 | | 1367.700 | ..........*X | 1367.700 | | 1275.800 | .........******..***.******..*X | 1888.100 | | 1887.500 | ..............*****X | 1887.500 | | 1131.000 | ........*X | 1131.000 | | 933.000 | ......******......***..******.....***..****X | 2688.400 | | 1904.100 | ..............*X | 1904.100 | | 1720.800 | .............******.**X | 1860.800 | | 1856.200 | ..............*****X | 1856.200 | | 1557.200 | ...........**.******..................*****.....**X | 4349.700 | | 1882.200 | ..............**X | 1882.200 | | 1349.400 | ..........******...........................**.......******..*X | 5762.800 | | 5691.400 | ..............................................*****X | 5691.400 | | 3190.100 | .........................**X | 3190.100 | | 2710.000 | .....................*****................**X | 4668.300 | | 4618.300 | .....................................*****X | 4618.300 | | 2272.100 | .................*X | 2272.100 | | 2096.000 | ................*****X | 2096.000 | | 2070.300 | ................*X | 2070.300 | | 1977.400 | ...............*****X | 1977.400 | | 1874.000 | ..............***X | 1874.000 | | 1848.400 | ..............****X | 1848.400 | | 988.800 | .......*X | 988.800 | | 706.700 | ....******.................**........******........***.*****X | 4708.600 | | 3029.500 | ........................*X | 3029.500 | | 2777.100 | ......................*****X | 2777.100 | | 1993.800 | ...............*X | 1993.800 | | 1610.100 | ............******.....***.*****X | 2340.100 | | 1877.100 | ..............*X | 1877.100 | | 1833.800 | ..............******......**.....******...............***.****X | 5094.800 | | 4121.300 | .................................**X | 4121.300 | | 3691.300 | .............................*****X | 3691.300 | | 2565.600 | ....................*X | 2565.600 | | 2313.200 | ..................******.....***.****X | 3010.400 | | 2938.400 | .......................****..*****..*X | 3423.800 | | 3315.000 | ..........................*****X | 3315.000 | | 2661.900 | .....................*X | 2661.900 | | 2432.700 | ...................******..***..******..**..******...*X | 3823.600 | | 3366.500 | ..........................*****X | 3366.500 | | 2102.100 | ................**X | 2102.100 | | 1853.700 | ..............*****X | 1853.700 | | 1745.900 | .............**..******.....**.******...***.******...**..*****X | 3742.600 | | 3732.800 | ..............................********..**X | 4017.100 | | 3670.000 | .............................*****X | 3670.000 | | 2031.200 | ...............*X | 2031.200 | | 1564.600 | ............******......**....*****X | 2831.100 | | 2448.200 | ...................**X | 2448.200 | | 2285.000 | ..................*****X | 2285.000 | | 1531.800 | ...........*X | 1531.800 | | 1124.300 | ........*****X | 1124.300 | | 1020.000 | .......**..******......................**.....*****X | 4518.800 | | 3056.400 | ........................**X | 3056.400 | | 2631.400 | ....................*****X | 2631.400 | | 1113.700 | ........*X | 1113.700 | | 975.400 | .......******.............**...******......***...*****X | 4058.500 | | 3260.600 | ..........................*X | 3260.600 | | 2660.400 | .....................******........**......*****X | 4419.800 | | 3858.000 | ...............................**X | 3858.000 | | 3296.200 | ..........................****X | 3296.200 | | 2486.500 | ...................**X | 2486.500 | | 2354.400 | ..................*****X | 2354.400 | | 1316.600 | .........*X | 1316.600 | | 1192.200 | ........******.........***.*****............***....*****X | 4286.000 | | 3674.800 | .............................*X | 3674.800 | | 3417.100 | ...........................*****X | 3417.100 | | 2802.500 | ......................**.******.......................***.....****X | 6322.200 | | 6103.300 | .................................................***.*****X | 6190.500 | | 4882.300 | .......................................*X | 4882.300 | | 4296.300 | ..................................*****X | 4296.300 | | 1928.000 | ...............**X | 1928.000 | | 1242.900 | .........****X | 1242.900 | | 985.800 | .......*********.....................**............******............................................**........*****X | 11285.200 | | 7831.100 | ................................................................**X | 7831.100 | | 6559.100 | .....................................................****X | 6559.100 | | 3611.600 | .............................**X | 3611.600 | | 2585.900 | ....................*****X | 2585.900 | | 890.600 | ......**..*******X | 1190.200 | | 1170.700 | ........*****X | 1170.700 | | 978.400 | .......**X | 978.400 | | 865.900 | ......*****.............................***..........*****X | 5591.100 | | 2242.200 | .................*X | 2242.200 | | 974.900 | .......*****X | 974.900 | | 515.700 | ...*X | 515.700 | | 310.300 | .*********.******.**..*****X | 737.800 | | 634.100 | ....*X | 634.100 | | 445.300 | ..******.........***.******..............**......******.........*******X | 5139.200 | | 2314.400 | ..................*X | 2314.400 | | 2074.100 | ................*****X | 2074.100 | | 1368.500 | ..........**X | 1368.500 | | 1054.900 | .......*****X | 1054.900 | | 1040.300 | .......**....******.................**....*****X | 3975.100 | | 2383.700 | ..................**X | 2383.700 | | 1399.400 | ..........*****..***....*****X | 2086.300 | | 1737.500 | .............*X | 1737.500 | | 1682.500 | ............******.......................**..............*****X | 6029.900 | | 4780.800 | ......................................**X | 4780.800 | | 3825.600 | ..............................*****X | 3825.600 | | 3210.000 | .........................*X | 3210.000 | | 2029.000 | ...............*****X | 2029.000 | | 1872.200 | ..............**.......*****X | 2736.400 | | 2472.800 | ...................**X | 2472.800 | | 2021.000 | ...............*****X | 2021.000 | | 1349.900 | ..........*X | 1349.900 | | 1045.500 | .......*****X | 1045.500 | | 731.600 | .....*X | 731.600 | | 281.700 | .******.......................***...........*****.......**.....X | 5843.300 | | 5689.100 | ..............................................*****X | 5689.100 | | 3041.700 | ........................*X | 3041.700 | | 2220.900 | .................*****X | 2220.900 | | 1757.700 | .............**X | 1757.700 | | 1656.800 | ............****X | 1656.800 | | 1639.700 | ............*********..********....*********.**..******..............**..*****X | 4654.900 | | 2667.000 | .....................**X | 2667.000 | | 1947.300 | ...............****X | 1947.300 | | 1347.000 | ..........**X | 1347.000 | | 1276.000 | .........******.....**....******.....*X | 2965.200 | | 2700.100 | .....................*****X | 2700.100 | | 2367.900 | ..................***..*****X | 2583.400 | | 2396.600 | ..................*X | 2396.600 | | 2219.200 | .................*****X | 2219.200 | | 1827.500 | ..............*X | 1827.500 | | 1685.600 | .............*****X | 1685.600 | | 1619.700 | ............***.******..*X | 1994.500 | | 1917.400 | ..............*****X | 1917.400 | | 1807.800 | ..............*X | 1807.800 | | 1802.600 | .............*****X | 1802.600 | | 1756.800 | .............**X | 1756.800 | | 1723.500 | .............*******.*******...**.*****X | 2386.800 | | 1899.600 | ..............**X | 1899.600 | | 1754.500 | .............******.....*X | 2341.600 | | 2322.800 | ..................*****X | 2322.800 | | 1764.300 | .............**.******.......***..****X | 2894.300 | | 1897.300 | ..............**X | 1897.300 | | 1245.600 | .........*****X | 1245.600 | | 785.900 | .....**.******....*X | 1439.600 | | 1279.000 | .........*****X | 1279.000 | | 822.700 | .....***..******......*X | 1745.900 | | 1660.500 | ............*****X | 1660.500 | | 1427.500 | ..........**...******....**X | 2195.600 | | 1993.200 | ...............****X | 1993.200 | | 968.700 | .......**X | 968.700 | | 760.700 | .....******.........**....******.....*X | 2981.600 | | 2817.800 | ......................*****X | 2817.800 | | 1760.400 | .............**X | 1760.400 | | 1713.100 | .............******..*X | 2010.300 | | 1962.200 | ...............******..**.*****X | 2383.700 | | 1439.000 | ..........**X | 1439.000 | | 1099.500 | ........*****X | 1099.500 | | 796.700 | .....*X | 796.700 | | 787.300 | .....******......********.***.******.......**.*****X | 2645.500 | | 2290.200 | ..................*X | 2290.200 | | 2238.400 | .................*****X | 2238.400 | | 2198.400 | .................**X | 2198.400 | | 2144.900 | ................*****........***....*****X | 3536.800 | | 1044.000 | .......*X | 1044.000 | | 416.300 | ..******...**.******.......*********.**.*****X | 1933.700 | | 1591.300 | ............*X | 1591.300 | | 1541.700 | ...........******.***.*****....**X | 2219.200 | | 2202.000 | .................*****X | 2202.000 | | 1854.000 | ..............*******X | 1880.500 | | 1571.700 | ............*X | 1571.700 | | 1439.300 | ..........******..***.******.......*******X | 2626.600 | | 782.500 | .....*X | 782.500 | | 466.900 | ..******...***.*****.**X | 978.300 | | 898.900 | ......*****X | 898.900 | | 460.900 | ..*X | 460.900 | | 416.900 | ..******....***.*****......*********.....**.******....**.*****X | 3047.300 | | 2735.000 | .....................**X | 2735.000 | | 2505.600 | ...................*****X | 2505.600 | | 2112.600 | ................*X | 2112.600 | | 2071.200 | ................*****X | 2071.200 | | 2020.100 | ...............**.*****X | 2080.000 | | 1491.700 | ...........**X | 1491.700 | | 1219.700 | .........****X | 1219.700 | | 707.600 | ....***.*************X | 735.000 | | 724.300 | .....**X | 724.300 | | 699.000 | ....****X | 699.000 | | 693.500 | ....**X | 693.500 | | 678.700 | ....*****X | 678.700 | | 640.000 | ....********.......*******X | 1527.900 | | 1370.900 | ..........********X | 1393.200 | | 627.900 | ....*X | 627.900 | | 554.300 | ...******...**.******.....*X | 1667.700 | | 1661.400 | ............*****X | 1661.400 | | 1476.900 | ...........***.*****X | 1575.900 | | 1165.100 | ........*X | 1165.100 | | 981.000 | .......******..********....********X | 1761.100 | | 1624.800 | ............********.....**...*****X | 2585.600 | | 1313.100 | .........**X | 1313.100 | | 916.100 | ......*****....**X | 1411.000 | | 1367.500 | ..........*****X | 1367.500 | | 1039.100 | .......**.******.********......**X | 1990.600 | | 1973.000 | ...............*****X | 1973.000 | | 1971.900 | ...............********...********..**X | 2637.000 | | 2635.400 | ....................*****X | 2635.400 | | 1879.300 | ..............**..******........*X | 3076.800 | | 2941.600 | .......................*****X | 2941.600 | | 2673.100 | .....................***...*******X | 3100.100 | | 2776.800 | ......................*****X | 2776.800 | | 1987.200 | ...............********.*X | 2112.300 | | 1890.500 | ..............*****X | 1890.500 | | 1195.100 | ........**X | 1195.100 | | 1152.600 | ........*****X | 1152.600 | | 743.800 | .....*X | 743.800 | | 727.400 | .....******......**...******.............**..*****X | 3666.900 | | 2186.800 | .................**X | 2186.800 | | 1810.600 | ..............*****X | 1810.600 | | 1338.100 | ..........*X | 1338.100 | | 1203.700 | .........******........**..******.***.******.......*X | 3511.300 | | 3506.900 | ............................*****X | 3506.900 | | 2182.400 | .................**.******......**X | 3078.500 | | 2583.300 | ....................*****.....***..*****X | 3453.400 | | 2229.000 | .................*X | 2229.000 | | 1924.800 | ...............*****X | 1924.800 | | 920.500 | ......**X | 920.500 | | 790.400 | .....****X | 790.400 | | 600.000 | ...***.******.*X | 734.700 | | 704.800 | ....*****X | 704.800 | | 619.800 | ....********.........***.****X | 1842.400 | | 1619.600 | ............***.******.********.........**.*****X | 3091.800 | | 2644.000 | ....................**X | 2644.000 | | 2411.500 | ...................******...**..*****X | 3022.700 | | 1803.300 | .............*X | 1803.300 | | 1642.400 | ............*****X | 1642.400 | | 794.500 | .....*X | 794.500 | | 680.300 | ....******...............***..*****X | 2700.600 | | 968.200 | .......*X | 968.200 | | 688.400 | ....******......**.*****X | 1507.500 | | 509.200 | ...**X | 509.200 | | 437.900 | ..******.....**.*****X | 1090.400 | | 600.800 | ...**.******........********...********X | 2023.200 | | 1826.700 | ..............*X | 1826.700 | | 1722.600 | .............******..**.******..........***..*****X | 3535.700 | | 3191.500 | .........................*X | 3191.500 | | 2963.000 | .......................*****X | 2963.000 | | 2812.200 | ......................**.*****X | 2888.500 | | 2463.000 | ...................*********........*X | 3429.400 | | 3347.000 | ..........................*****X | 3347.000 | | 2787.200 | ......................*X | 2787.200 | | 2764.300 | .....................*****X | 2764.300 | | 2651.900 | .....................**X | 2651.900 | | 2565.900 | ....................****X | 2565.900 | | 1926.900 | ...............*********..........*******X | 3215.700 | | 2267.500 | .................*X | 2267.500 | | 2081.200 | ................*****X | 2081.200 | | 1414.200 | ..........**X | 1414.200 | | 1383.000 | ..........*****X | 1383.000 | | 1280.300 | .........*X | 1280.300 | | 1166.800 | ........*****X | 1166.800 | | 1067.400 | .......**.******........X | 2076.400 | +----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 03:54:14,734 | test | INFO | pool-3-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****............................................................................***........................*****X | 753.800 | | 189.000 | ........................*X | 189.000 | | 15.300 | .*****X | 15.300 | | 3.500 | *X | 3.500 | | 0.000 | *********************************************************************************************************....**..*****X | 49.900 | | 12.700 | *X | 12.700 | | 0.000 | *******************************************************....***..*****X | 45.900 | | 12.000 | *X | 12.000 | | 0.000 | ***********************..*X | 15.300 | | 11.300 | *****X | 11.300 | | 0.000 | ***************************..*****X | 15.300 | | 8.100 | X | 8.100 | | 4.000 | X | 4.000 | | 0.000 | **************..**..*****X | 30.600 | | 8.000 | **X | 8.000 | | 4.000 | *****......***..*****X | 66.200 | | 57.800 | ......*X | 57.800 | | 40.800 | ....*****X | 40.800 | | 15.300 | .***************X | 15.300 | | 4.000 | **X | 4.000 | | 0.000 | **********************...***.****X | 34.500 | | 26.700 | ..X | 26.700 | | 9.000 | *X | 9.000 | | 0.000 | **************..*X | 15.300 | | 11.300 | *****X | 11.300 | | 0.000 | *********************************.....***...****X | 61.200 | | 16.000 | .**X | 16.000 | | 0.000 | **************....**..*****X | 49.800 | | 35.400 | ...*X | 35.400 | | 19.200 | .*****X | 19.200 | | 5.000 | **X | 5.000 | | 0.000 | *******************************************************...*X | 26.800 | | 26.600 | ..*****X | 26.600 | | 15.100 | .*********............**....*****X | 137.700 | | 35.600 | ...*X | 35.600 | | 8.000 | ******...**X | 30.600 | | 22.600 | .****X | 22.600 | | 7.400 | X | 7.400 | | 0.000 | *********************************************************....*****..***X | 45.900 | | 12.000 | *X | 12.000 | | 0.000 | **************.**..******.............***.****X | 132.500 | | 36.300 | ...**X | 36.300 | | 15.300 | .*****X | 15.300 | | 4.000 | *X | 4.000 | | 0.000 | *************************.******......*******X | 57.200 | | 23.300 | ..*X | 23.300 | | 19.300 | .******..**X | 30.600 | | 22.600 | .*****X | 22.600 | | 11.300 | **.******..*X | 30.600 | | 26.600 | ..*****X | 26.600 | | 4.000 | **X | 4.000 | | 0.000 | *****....***..****X | 45.900 | | 38.100 | ....X | 38.100 | | 34.600 | ...*X | 34.600 | | 30.600 | ...*****X | 30.600 | | 8.000 | *X | 8.000 | | 0.000 | ******....**..*****X | 49.800 | | 13.000 | **X | 13.000 | | 0.000 | **************..**..******....*X | 61.100 | | 60.100 | ......****X | 60.100 | | 60.100 | ......X | 60.100 | | 12.000 | **X | 12.000 | | 0.000 | **************.**.*****X | 19.200 | | 16.300 | .**.******...**X | 45.900 | | 41.900 | ....*****X | 41.900 | | 19.300 | .*X | 19.300 | | 11.300 | ********.********X | 19.300 | | 11.300 | ****X | 11.300 | | 0.000 | *********..**...******......**.*****X | 90.700 | | 31.100 | ...**X | 31.100 | | 11.300 | ****X | 11.300 | | 7.800 | X | 7.800 | | 0.000 | ********......**...*****X | 69.000 | | 40.600 | ....*X | 40.600 | | 30.600 | ...**************X | 34.500 | | 31.600 | ...********..*X | 49.800 | | 44.800 | ....*****X | 44.800 | | 19.300 | .**X | 19.300 | | 15.300 | .**************...**..*****X | 53.800 | | 50.900 | .....**X | 50.900 | | 41.900 | ....****X | 41.900 | | 33.500 | ...*********...**.*****X | 65.100 | | 28.300 | ..*X | 28.300 | | 15.300 | .*****X | 15.300 | | 4.000 | *X | 4.000 | | 0.000 | ************************************************.........**....*****X | 99.600 | | 25.800 | ..**X | 25.800 | | 0.000 | **************....***..****X | 49.800 | | 24.100 | ..**X | 24.100 | | 15.300 | .**************.*X | 23.200 | | 14.200 | *****X | 14.200 | | 0.000 | *************************..**X | 15.300 | | 11.300 | *****X | 11.300 | | 0.000 | *************************************************....***..*****X | 49.800 | | 24.300 | ..*X | 24.300 | | 19.300 | .******..*X | 30.600 | | 26.600 | ..*****X | 26.600 | | 4.000 | **X | 4.000 | | 0.000 | ***********************************************.**.******.**.*****X | 34.600 | | 23.300 | ..**X | 23.300 | | 11.300 | *****X | 11.300 | | 0.000 | ****************.***.*************X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | *******************************************************************************************************************************************************.******...*X | 34.500 | | 25.700 | ..*****X | 25.700 | | 0.000 | *********************************.***.*************X | 15.300 | | 4.000 | *X | 4.000 | | 0.000 | ******.***.*****X | 15.300 | | 4.000 | *X | 4.000 | | 0.000 | *******************************.**.*****X | 19.200 | | 5.000 | **X | 5.000 | | 0.000 | ******************************************************************..******........*X | 76.400 | | 63.400 | .......*****X | 63.400 | | 4.000 | **X | 4.000 | | 0.000 | **********************************************************************************************************************************************************************************************************************************************************************...**..*****X | 38.400 | | 10.000 | **X | 10.000 | | 0.000 | **********************************************************************************************************************************************************************************************************************************************************************.**.X | 15.300 | | 11.300 | *****X | 11.300 | | 0.000 | *******************************************************************************************.***.*****X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | ******.**.*****X | 19.200 | | 5.000 | **X | 5.000 | | 0.000 | **************...**..*****X | 38.400 | | 10.000 | **X | 10.000 | | 0.000 | ******************************************************************..******......**X | 65.100 | | 48.500 | .....*****X | 48.500 | | 0.000 | ******************************************************************************************************************************.**.*****X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | *****************************************************************.....**..*****X | 57.600 | | 15.000 | *X | 15.000 | | 0.000 | *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************.***.*****X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | ******...**.*****X | 34.500 | | 9.000 | **X | 9.000 | | 0.000 | ******************************************************************************************************************......**...*****X | 69.000 | | 17.800 | .*X | 17.800 | | 0.000 | **************************************************************************************************************************.***.*****X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | ***********************************************************************************************************************************************************************************************.******....**X | 44.800 | | 41.800 | ....*****X | 41.800 | | 5.000 | *X | 5.000 | | 0.000 | ********************************************************.**.*****X | 15.300 | | 4.000 | **X | 4.000 | | 0.000 | ******.**.*****X | 15.300 | | 7.200 | X | 7.200 | | 3.800 | *X | 3.800 | | 0.000 | **********************...***.*****X | 34.500 | | 8.800 | *X | 8.800 | | 0.000 | *******************************..***..********.******..*X | 53.500 | | 34.100 | ...******..***..*****X | 65.100 | | 16.400 | .*X | 16.400 | | 0.000 | *******************************************************************************************************************.**.*****X | 15.300 | | 4.000 | *X | 4.000 | | 0.000 | ***************************************************************************************************************************.***.******.**.*****X | 34.500 | | 9.000 | **X | 9.000 | | 0.000 | *****************************************************************************************..**********X | 15.300 | | 11.300 | *****X | 11.300 | | 0.000 | *****************************************************************************************************************************************.******...*X | 30.600 | | 22.600 | .*****X | 22.600 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************.***.*****X | 15.300 | | 4.000 | *X | 4.000 | | 0.000 | **************************************************.******...*X | 34.500 | | 25.700 | ..*****X | 25.700 | | 0.000 | ***********************************************************.**..*****X | 23.000 | | 20.300 | .*X | 20.300 | | 11.500 | *****X | 11.500 | | 0.000 | *************************.***.*****X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | *************************************************************************************************...***.*****...***..*****X | 69.000 | | 17.800 | .*X | 17.800 | | 0.000 | ******.**..******.**X | 34.500 | | 30.700 | ...****X | 30.700 | | 5.000 | **X | 5.000 | | 0.000 | ******.**..******....*X | 53.700 | | 44.900 | ....*****X | 44.900 | | 30.500 | ...*******X | 34.500 | | 20.300 | .**X | 20.300 | | 15.300 | .******.**.*****X | 34.500 | | 8.800 | *X | 8.800 | | 0.000 | ***************...**..*****X | 43.400 | | 29.200 | ..*X | 29.200 | | 14.200 | *****X | 14.200 | | 0.000 | *****************************************.***.*****X | 19.200 | | 5.000 | *X | 5.000 | | 3.800 | ******..**********X | 19.200 | | 14.200 | *****X | 14.200 | | 0.000 | ****************.***.*************X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | ***********************...**.*****X | 30.600 | | 7.600 | **X | 7.600 | | 0.000 | ***********************************************...***.*****X | 34.500 | | 8.800 | *X | 8.800 | | 0.000 | **************.***.******..*******X | 34.500 | | 8.800 | ***X | 8.800 | | 0.000 | ****.***.*************X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | ******...*****.***X | 34.500 | | 8.800 | *X | 8.800 | | 3.800 | ******.....**.******.**X | 53.700 | | 48.700 | .....****X | 48.700 | | 8.800 | **X | 8.800 | | 5.000 | ******....*X | 30.700 | | 29.500 | ..*****X | 29.500 | | 3.800 | **.******...**X | 34.500 | | 25.700 | ..*****X | 25.700 | | 0.000 | ****************.......***..*****X | 72.900 | | 33.200 | ...*X | 33.200 | | 19.200 | .******...**.*****X | 45.900 | | 22.900 | ..**X | 22.900 | | 15.300 | .*************X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | ***********************..*X | 15.300 | | 11.500 | *****X | 11.500 | | 0.000 | *****************..***X | 15.300 | | 11.500 | ***X | 11.500 | | 0.000 | *******************.******....*X | 44.800 | | 42.200 | ....*****X | 42.200 | | 19.200 | .********X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | ******.**.*****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | **************...**.******..***.****X | 53.700 | | 13.800 | **X | 13.800 | | 0.000 | **********************.***.*****X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | **************.***.*************X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | ********************************************************..**X | 19.200 | | 14.200 | *****X | 14.200 | | 0.000 | ********.***.*****X | 19.200 | | 5.000 | *X | 5.000 | | 3.800 | ******..*X | 15.300 | | 11.500 | *****X | 11.500 | | 0.000 | ***..******........*X | 80.400 | | 60.200 | ......*****X | 60.200 | | 0.000 | *************************.**.******..............***...*****X | 149.500 | | 46.700 | .....*X | 46.700 | | 15.300 | .*****X | 15.300 | | 3.800 | ********..**X | 15.300 | | 11.500 | *****X | 11.500 | | 0.000 | ************************************************************************************...**.*****X | 30.600 | | 7.600 | *X | 7.600 | | 0.000 | ******...***..*****X | 38.400 | | 10.000 | *X | 10.000 | | 0.000 | ************************************************....**..*****X | 49.800 | | 12.600 | **X | 12.600 | | 5.000 | *****..**X | 19.200 | | 14.200 | *****X | 14.200 | | 0.000 | *************************...**.*****X | 30.600 | | 15.200 | .X | 15.200 | | 7.600 | X | 7.600 | | 0.000 | ***********************.**.*****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | ******...**.*****X | 34.500 | | 8.800 | *X | 8.800 | | 0.000 | ******.**.*****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | **************.**.********X | 19.100 | | 11.500 | ******...**..*****X | 45.900 | | 11.400 | *X | 11.400 | | 0.000 | ********************************************************......***.*****X | 57.400 | | 11.400 | *X | 11.400 | | 0.000 | ******.**.*****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | **************..*X | 15.300 | | 11.500 | *****X | 11.500 | | 0.000 | ********************************************************************************************..**X | 15.300 | | 11.500 | *****X | 11.500 | | 0.000 | ******************************************..*X | 15.300 | | 11.500 | *********.****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | *******************************.**.*****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | **************...**.*****X | 30.600 | | 7.600 | **X | 7.600 | | 0.000 | ***********************.**.*****X | 19.200 | | 5.000 | *X | 5.000 | | 0.000 | *****************************************************************************************************************************************************.***.*****X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | ******.**.*****X | 15.300 | | 3.800 | **X | 3.800 | | 0.000 | **************..*X | 15.300 | | 11.500 | *****X | 11.500 | | 0.000 | *****************************************************************************************************************************************************************************************.**.*****X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | *******************************************************************************************************************...***.*****X | 30.600 | | 7.600 | *X | 7.600 | | 0.000 | ******.***.*************X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | ***************************************************************************************************************************************************************************************************************X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 03:54:14,792 | test | INFO | pool-3-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****......................................................***.......................****X | 60.800 | | 49.600 | ..............................................................X | 49.600 | | 18.400 | ......................*X | 18.400 | | 0.000 | ******************************************************************************************************************......**...*****X | 7.600 | | 2.300 | .*X | 2.300 | | 0.000 | ***********************************************************************************..******.......*X | 7.600 | | 5.900 | ......*****X | 5.900 | | 0.000 | *************************....**...........******X | 11.900 | | 6.700 | .......X | 6.700 | | 3.400 | ...******...........**....*****X | 15.200 | | 3.000 | ..*X | 3.000 | | 0.000 | *********.*****................***....******.................*X | 29.800 | | 29.400 | ....................................*****X | 29.400 | | 4.100 | ....*X | 4.100 | | 0.000 | ***************************************...............***....*****X | 15.200 | | 3.000 | ..*X | 3.000 | | 0.000 | ********.******................*******X | 13.700 | | 1.500 | **X | 1.500 | | 0.000 | **************.......**..*****X | 7.600 | | 1.500 | **X | 1.500 | | 0.000 | *****...............***......*****X | 16.700 | | 10.600 | ............*X | 10.600 | | 6.100 | ......*****X | 6.100 | | 0.000 | ********......................**.......*****X | 22.800 | | 12.600 | ...............*X | 12.600 | | 4.800 | .....X | 4.800 | | 0.000 | ********..******....*...X | 7.600 | | 5.800 | ......*****X | 5.800 | | 0.000 | ************************************..******...........................**.........*****X | 30.400 | | 14.500 | .................*X | 14.500 | | 5.300 | .....********...*****X | 7.600 | | 2.300 | .**X | 2.300 | | 0.000 | ************************..******.......**X | 7.600 | | 5.300 | .....****X | 5.300 | | 0.000 | *********......**...*****X | 7.600 | | 2.600 | ..**X | 2.600 | | 0.000 | ********...******.................................****.........***X | 35.600 | | 9.600 | ...........*X | 9.600 | | 0.000 | **************........***.........*****.............................***...*****X | 39.100 | | 20.000 | ........................*X | 20.000 | | 15.200 | ..................*****X | 15.200 | | 4.800 | .....*X | 4.800 | | 0.000 | *************************...******......*X | 7.600 | | 5.200 | .....********......********X | 10.000 | | 5.200 | .....********.........******.......*X | 17.600 | | 10.400 | ............*********.........****X | 17.600 | | 12.400 | ..............**X | 12.400 | | 5.200 | .....********...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ******......***...****X | 7.600 | | 2.400 | ..**X | 2.400 | | 0.000 | ********...******..........................**......*****X | 28.000 | | 7.200 | ........**X | 7.200 | | 0.000 | *****......***...******.............**.......*************X | 22.800 | | 7.200 | ........**X | 7.200 | | 2.400 | ..******...................**................******......*X | 35.200 | | 20.800 | .........................*****X | 20.800 | | 5.200 | .....***...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ******.............**......******....................**...........................******..................................**X | 78.400 | | 66.400 | ...................................................................................*****X | 66.400 | | 24.800 | ..............................*X | 24.800 | | 10.400 | ............*****X | 10.400 | | 0.000 | ********......***...**************.......**...******.......***...****X | 22.800 | | 17.600 | .....................**X | 17.600 | | 15.200 | ..................*****X | 15.200 | | 4.800 | .....*X | 4.800 | | 0.000 | **************.................................***...............*****X | 38.000 | | 35.000 | ...........................................*X | 35.000 | | 32.400 | ........................................****X | 32.400 | | 32.400 | ........................................X | 32.400 | | 9.400 | ..........*X | 9.400 | | 0.000 | ******.............**......*****X | 15.200 | | 4.800 | .....*X | 4.800 | | 2.400 | ..******......**X | 7.600 | | 5.200 | .....*****X | 5.200 | | 0.000 | ****************......***...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | **************...............................................***.......................*****X | 55.600 | | 29.000 | ...................................*X | 29.000 | | 12.800 | ...............********...*****X | 15.200 | | 10.000 | ...........**X | 10.000 | | 9.800 | ...........******..........................**...*****X | 32.200 | | 18.500 | ......................X | 18.500 | | 12.200 | ..............X | 12.200 | | 7.600 | ........*****X | 7.600 | | 2.400 | ..**X | 2.400 | | 0.000 | ******......**...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ******......***............******.......................X | 32.400 | | 32.400 | ........................................X | 32.400 | | 25.400 | ...............................*****X | 25.400 | | 2.400 | ..*X | 2.400 | | 0.000 | ***************************************......******...**X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ************************************************************************.............***......****X | 15.200 | | 4.800 | .....**X | 4.800 | | 0.000 | ******************************************************************************************************************************************************************************************************.............**...............******.........**X | 29.600 | | 17.800 | .....................*****X | 17.800 | | 0.000 | *********************************......**...**************X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | *****************************************************************************************************************************.....******..........................*X | 24.600 | | 22.200 | ...........................*****X | 22.200 | | 2.200 | .**X | 2.200 | | 0.000 | ***************************************************************......***......******...*X | 12.000 | | 7.400 | ........*****X | 7.400 | | 0.000 | *************************......**...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ***************************************************************************************************************************************************......**...*****X | 7.600 | | 2.400 | ..**X | 2.400 | | 0.000 | ****************************************************************......**...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ******************************************************************************************************************************************......**...*****X | 7.600 | | 2.400 | ..**X | 2.400 | | 0.000 | *********************************...******....*..X | 7.600 | | 5.200 | .....*****X | 5.200 | | 0.000 | ******************************************************************************************************************************************************......***...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | **************************************************...........******................................*X | 34.400 | | 25.200 | ...............................*****X | 25.200 | | 0.000 | *********************************************************************************************************************************...*****....*..*X | 7.600 | | 5.200 | .....*****X | 5.200 | | 0.000 | **********************************************************.........................**.........*****X | 26.800 | | 6.800 | .......**X | 6.800 | | 0.000 | *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************......***...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ******...................**..........*****X | 22.800 | | 7.200 | ........**X | 7.200 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************......**......******................................***......*****X | 39.600 | | 14.400 | .................*X | 14.400 | | 7.600 | ........*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ***************************************************************************************************************************************************************************************..******................***.*****X | 15.000 | | 7.600 | ........*******X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ************************************************......***...**********************..*.*X | 9.800 | | 9.600 | ...........*****X | 9.600 | | 2.200 | .*X | 2.200 | | 0.000 | ***************......**...******...................**........*****X | 29.000 | | 16.400 | ...................**X | 16.400 | | 7.400 | ........*****X | 7.400 | | 0.000 | ****************......***...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 2.200 | .******..........**X | 9.600 | | 7.400 | ........*****X | 7.400 | | 0.000 | ***********************************************************************************************************************************************************************......................***.........*****X | 24.800 | | 7.000 | .......*X | 7.000 | | 0.000 | ***************************************......................***.........*****X | 24.800 | | 7.000 | .......*X | 7.000 | | 0.000 | ***************************************.........***...*****X | 9.600 | | 2.200 | .*X | 2.200 | | 0.000 | ************************************************************************************.....******..............*X | 15.200 | | 11.000 | .............*****X | 11.000 | | 0.000 | *************************************************************************************************************************************************.....******................**X | 17.200 | | 12.600 | ...............*****X | 12.600 | | 0.000 | *********************************************************************************************************************..................**......*****X | 19.200 | | 4.400 | ....**X | 4.400 | | 0.000 | **************......**...*****X | 7.600 | | 2.400 | ..**X | 2.400 | | 0.000 | *************************************************************************************************************************************************************************************************************************************************************************************************.............***......*****X | 15.200 | | 4.800 | .....*X | 4.800 | | 0.000 | *************************.....******...................*X | 19.200 | | 14.800 | .................*****X | 14.800 | | 0.000 | ***********.....******................*X | 17.200 | | 12.600 | ...............*****X | 12.600 | | 0.000 | ***********************************************************.......**..*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | *******************************.........***...*****X | 9.600 | | 2.200 | .*X | 2.200 | | 0.000 | **************......***...****X | 7.600 | | 2.400 | ..**X | 2.400 | | 0.000 | ********************************************************......**...*****X | 7.600 | | 2.400 | ..*X | 2.400 | | 0.000 | ******............................***........****X | 28.800 | | 6.600 | .......**X | 6.600 | | 0.000 | **************..................**.............******.........................**X | 44.000 | | 38.200 | ...............................................****X | 38.200 | | 25.000 | ..............................X | 25.000 | | 19.200 | .......................*******X | 19.200 | | 11.800 | ..............**...******...........*X | 23.200 | | 18.000 | .....................*****X | 18.000 | | 7.600 | ........********X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | ******.........**...*****X | 9.600 | | 2.200 | .*X | 2.200 | | 0.000 | ************************************************************************..............***.....*****X | 15.200 | | 4.200 | ....*X | 4.200 | | 0.000 | ***************************************................***.....*****X | 17.200 | | 4.600 | ....*X | 4.600 | | 0.000 | ********************************************************.........**...*****X | 9.600 | | 2.200 | .**X | 2.200 | | 0.000 | **********************..................***......*****X | 19.200 | | 4.400 | ....**...******..........................*X | 26.800 | | 20.000 | ........................*****X | 20.000 | | 5.800 | ......***..*************X | 7.600 | | 1.800 | .********...................********X | 17.000 | | 15.400 | ..................**..*****X | 17.200 | | 4.000 | ....*X | 4.000 | | 0.000 | ******..............*****.....***X | 15.200 | | 3.600 | ...*X | 3.600 | | 2.200 | .******....................................**.....*****X | 34.200 | | 19.400 | .......................**X | 19.400 | | 17.200 | ....................****X | 17.200 | | 4.000 | ....**X | 4.000 | | 0.000 | ******.............................................**.............*****X | 46.000 | | 10.600 | ............*X | 10.600 | | 0.000 | *******************..****.................*X | 15.400 | | 15.000 | ..................******................***........*****X | 34.400 | | 13.800 | ................*X | 13.800 | | 11.500 | .............******...................*X | 26.300 | | 22.900 | ............................*****X | 22.900 | | 2.300 | .**X | 2.300 | | 0.000 | ***************************************................**.....******.....................***......****X | 38.400 | | 8.800 | ..........**X | 8.800 | | 2.200 | .******................................................****........***X | 45.800 | | 8.800 | ..........*X | 8.800 | | 0.000 | ******.........***...*****X | 9.600 | | 2.200 | .*X | 2.200 | | 1.800 | .******..........................**...*****X | 25.000 | | 4.400 | ....**X | 4.400 | | 0.000 | **************.........**...******..............***.....****X | 24.800 | | 6.400 | .......**X | 6.400 | | 2.200 | .******.................*X | 15.400 | | 15.000 | ..................*****X | 15.000 | | 12.800 | ...............***...****X | 15.200 | | 10.000 | ...........**X | 10.000 | | 7.600 | ........*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | **************.........***...*****X | 9.600 | | 2.200 | .*X | 2.200 | | 0.000 | ***********************.......**..*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | ************************************************.....***.......******..............*X | 20.800 | | 15.000 | ..................*****X | 15.000 | | 5.800 | ......**..**************X | 7.600 | | 1.800 | .********.........................***.*****........***........*****X | 34.600 | | 17.200 | ....................*X | 17.200 | | 7.400 | ........*****X | 7.400 | | 0.000 | *************************.......***..********...******..*X | 11.400 | | 7.400 | ........*****X | 7.400 | | 0.000 | ***..******.......*X | 7.600 | | 5.800 | ......*****X | 5.800 | | 0.000 | ***************************************************************************................***........*******X | 19.000 | | 15.400 | ..................*****X | 15.400 | | 9.600 | ...........********.......***..*****X | 17.200 | | 4.000 | ....*X | 4.000 | | 0.000 | ******..............***.....*****X | 15.200 | | 3.600 | ...*X | 3.600 | | 0.000 | *******************************................**.....*****X | 17.200 | | 4.000 | ....*X | 4.000 | | 1.800 | .******.......**X | 7.600 | | 5.800 | ......*****X | 5.800 | | 0.000 | ***************************..******.....*..X | 7.600 | | 5.800 | ......*****X | 5.800 | | 0.000 | **********************************................**.....*****X | 17.200 | | 4.000 | ....*X | 4.000 | | 0.000 | ******...............................**..........*****X | 32.400 | | 7.600 | ........**X | 7.600 | | 0.000 | **************..............**.....*****X | 15.200 | | 3.600 | ...**X | 3.600 | | 0.000 | ******.......................**........*****X | 24.800 | | 6.400 | .......*X | 6.400 | | 0.000 | ***********************.........**...*****X | 9.600 | | 8.000 | .........**X | 8.000 | | 7.600 | ........*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | **************.......***..*****X | 7.600 | | 1.800 | .**...******................*X | 17.200 | | 13.200 | ...............*****X | 13.200 | | 0.000 | ********.....*.........**........*****X | 17.400 | | 13.200 | ...............*X | 13.200 | | 7.400 | ........*****X | 7.400 | | 0.000 | *************************......................***.......****X | 22.800 | | 5.400 | .....**X | 5.400 | | 0.000 | ********************************************************..............**.......*****X | 17.000 | | 11.200 | .............**X | 11.200 | | 5.800 | ......*****X | 5.800 | | 0.000 | ***************************..******...............*******X | 13.400 | | 7.600 | ........********X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | *******************************................***.....*****X | 17.200 | | 4.000 | ....*X | 4.000 | | 0.000 | ***************.......**....*******X | 9.400 | | 5.800 | ......*****X | 5.800 | | 0.000 | **************************************************************************************************************************************.......***..*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | **************.......***..*****X | 7.600 | | 1.800 | .********.............................***.....*****X | 28.600 | | 5.400 | .....*X | 5.400 | | 0.000 | ********..******.......*X | 7.600 | | 5.800 | ......*********..*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | ***************************************.............................***.........*****X | 30.400 | | 7.200 | ........*X | 7.200 | | 0.000 | ******..............***.....*****X | 15.200 | | 3.600 | ...*X | 3.600 | | 0.000 | *******************************..............**.....*****X | 15.200 | | 3.600 | ...**X | 3.600 | | 0.000 | ******.......**..*****X | 7.600 | | 1.800 | .**X | 1.800 | | 0.000 | ********************************************************.......**..**************X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | ***********************.......**..*****X | 7.600 | | 1.800 | .*X | 1.800 | | 0.000 | *************************************************************************.......***..******........**..*****X | 15.200 | | 3.600 | ...*X | 3.600 | | 0.000 | ***************..............**.....*****X | 15.200 | | 3.600 | ...**X | 3.600 | | 0.000 | *****************************************************************************************.......***..*****X | 7.600 | | 1.800 | .********.........................**.*****X | 22.600 | | 3.600 | ...**X | 3.600 | | 0.000 | *****************************************************************************************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 03:54:16,055 | test | INFO | pool-3-thread-10 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------+----------+ | 2076.400 | ...................................................................................................X | 2076.400 | +----------+------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 03:54:16,207 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.105.206']} 2022-02-02 03:59:16,601 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 03:59:29,506 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 03:59:29,506 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 03:59:38,589 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 03:59:38,589 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 03:59:47,799 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 03:59:47,799 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 04:00:02,775 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 92100 | 7.81 GiB | 292.31 MiB | 372.15 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 3.91 GiB | 447.72 MiB | 372.78 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 15010750 | 2.00 GiB | 1.08 GiB | 30.54 GiB | 33.7137318255 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 04:00:03,295 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 04:00:03,295 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 04:00:09,746 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 04:00:09,746 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 04:00:16,092 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 32% 2022-02-02 04:00:16,094 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 12% 2022-02-02 04:01:08,729 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 04:01:09,161 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 04:01:09,161 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 04:01:39,309 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 04:01:39,721 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 04:01:39,721 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 04:02:09,832 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 04:02:10,244 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 04:02:10,246 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 04:04:11,209 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'dfe1f5a52717f2f48c47547157144f5c', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c710ad17d6276c00f01d73f7f64f25fb', u'status': u'notRunning'} - rebalance failed 2022-02-02 04:04:11,233 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 04:04:11,233 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643803450443L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:04:10.443Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default","bucket2"]}}}]}.\nRebalance Operation Id = feb6cab8816de78b77e3b1f0bead2de9'} 2022-02-02 04:04:11,233 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643803450442L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:04:10.442Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default",\n "bucket2"]}}}]\n'} 2022-02-02 04:04:11,234 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643803330439L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:02:10.439Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = feb6cab8816de78b77e3b1f0bead2de9"} 2022-02-02 04:04:11,234 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643803330431L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T04:02:10.431Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\']'} 2022-02-02 04:04:11,234 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643803329821L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:02:09.821Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.105.36\' for deletion'} 2022-02-02 04:04:11,236 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643803324539L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:02:04.539Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 04:04:11,236 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643803319136L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:01:59.136Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.105.164\' for deletion'} 2022-02-02 04:04:11,236 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643803299196L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:01:39.196Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.105.206\' for deletion'} 2022-02-02 04:04:11,236 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643803298482L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:01:38.482Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 04:04:11,236 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643803297851L, u'shortText': u'message', u'serverTime': u'2022-02-02T04:01:37.851Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.105.33\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress 2022-02-02 04:04:11,237 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'dfe1f5a52717f2f48c47547157144f5c', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c710ad17d6276c00f01d73f7f64f25fb', u'status': u'notRunning'} - rebalance failed return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'dfe1f5a52717f2f48c47547157144f5c', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c710ad17d6276c00f01d73f7f64f25fb', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- During the test, Remote Connections: 241, Disconnections: 241 SDK Connections: 30, Disconnections: 30 Ran 1 test in 16119.814s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_1 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,bucket_spec=magma_dgm.10_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=hard_failover_and_rebalance_out_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_2', 'nodes_init': '5', 'GROUP': 'hard_failover_and_rebalance_out_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.10_percent_dgm.5_node_2_replica_magma_512', 'case_number': 2, 'randomize_value': 'True'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 04:04:17,351 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:04:17,755 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:05:21,361 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 63.3600001335 seconds to complete 2022-02-02 04:05:21,368 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 04:05:26,441 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xd6f43 type=client open_count=1 channel=[id: 0x9efde4c9, 0.0.0.0/0.0.0.0:58077] timeout=300.0>) 2022-02-02 04:05:30,224 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xd6f44 type=client open_count=1 channel=[id: 0xe8e03ed0, 0.0.0.0/0.0.0.0:49976] timeout=300.0>) 2022-02-02 04:05:32,236 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xd6f45 type=client open_count=1 channel=[id: 0xddbd162f, 0.0.0.0/0.0.0.0:53188] timeout=300.0>) 2022-02-02 04:05:34,246 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xd6f46 type=client open_count=1 channel=[id: 0xc137f739, 0.0.0.0/0.0.0.0:39826] timeout=300.0>) 2022-02-02 04:05:36,257 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xd6f47 type=client open_count=1 channel=[id: 0xcb0aa24d, 0.0.0.0/0.0.0.0:36617] timeout=300.0>) 2022-02-02 04:05:38,269 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xd6f48 type=client open_count=1 channel=[id: 0xbe8c956f, 0.0.0.0/0.0.0.0:47891] timeout=300.0>) 2022-02-02 04:05:43,862 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xd6f49 type=client open_count=1 channel=[id: 0x70098270, 0.0.0.0/0.0.0.0:46519] timeout=300.0>) 2022-02-02 04:05:45,872 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xd6f4a type=client open_count=1 channel=[id: 0x34a5ae88, 0.0.0.0/0.0.0.0:35787] timeout=300.0>) 2022-02-02 04:05:47,884 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xd6f4b type=client open_count=1 channel=[id: 0xb693b2cf, 0.0.0.0/0.0.0.0:42313] timeout=300.0>) 2022-02-02 04:05:49,894 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xd6f4c type=client open_count=1 channel=[id: 0xaf844695, 0.0.0.0/0.0.0.0:34417] timeout=300.0>) 2022-02-02 04:05:51,904 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xd6f4d type=client open_count=1 channel=[id: 0xeee48893, 0.0.0.0/0.0.0.0:39125] timeout=300.0>) 2022-02-02 04:05:57,413 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xd6f4e type=client open_count=1 channel=[id: 0xd79e4202, 0.0.0.0/0.0.0.0:37713] timeout=300.0>) 2022-02-02 04:05:59,424 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xd6f4f type=client open_count=1 channel=[id: 0x2fc5a8d2, 0.0.0.0/0.0.0.0:55498] timeout=300.0>) 2022-02-02 04:06:01,436 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xd6f50 type=client open_count=1 channel=[id: 0xa0cddf97, 0.0.0.0/0.0.0.0:49551] timeout=300.0>) 2022-02-02 04:06:03,447 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xd6f51 type=client open_count=1 channel=[id: 0x0c79dc69, 0.0.0.0/0.0.0.0:44125] timeout=300.0>) 2022-02-02 04:06:05,459 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xd6f52 type=client open_count=1 channel=[id: 0xb4c181a0, 0.0.0.0/0.0.0.0:46389] timeout=300.0>) 2022-02-02 04:06:10,913 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xd6f53 type=client open_count=1 channel=[id: 0xd1f76e8d, 0.0.0.0/0.0.0.0:38390] timeout=300.0>) 2022-02-02 04:06:12,926 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xd6f54 type=client open_count=1 channel=[id: 0x1ea1a522, 0.0.0.0/0.0.0.0:50429] timeout=300.0>) 2022-02-02 04:06:14,937 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xd6f55 type=client open_count=1 channel=[id: 0x8ef877b0, 0.0.0.0/0.0.0.0:51641] timeout=300.0>) 2022-02-02 04:06:16,947 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xd6f56 type=client open_count=1 channel=[id: 0x9e27bbba, 0.0.0.0/0.0.0.0:37581] timeout=300.0>) 2022-02-02 04:06:18,960 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xd6f57 type=client open_count=1 channel=[id: 0x970edfd2, 0.0.0.0/0.0.0.0:58267] timeout=300.0>) 2022-02-02 04:06:27,061 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xd6f58 type=client open_count=1 channel=[id: 0xdca2da5e, 0.0.0.0/0.0.0.0:55377] timeout=300.0>) 2022-02-02 04:06:29,073 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xd6f59 type=client open_count=1 channel=[id: 0x999367de, 0.0.0.0/0.0.0.0:35325] timeout=300.0>) 2022-02-02 04:06:31,085 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xd6f5a type=client open_count=1 channel=[id: 0xce147ca4, 0.0.0.0/0.0.0.0:50016] timeout=300.0>) 2022-02-02 04:06:33,095 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xd6f5b type=client open_count=1 channel=[id: 0x5af66d8b, 0.0.0.0/0.0.0.0:44243] timeout=300.0>) 2022-02-02 04:06:35,107 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f5c type=client open_count=1 channel=[id: 0x517bdb42, 0.0.0.0/0.0.0.0:49893] timeout=300.0>) 2022-02-02 04:06:40,687 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f5d type=client open_count=1 channel=[id: 0xc1130e3c, 0.0.0.0/0.0.0.0:51168] timeout=300.0>) 2022-02-02 04:06:42,698 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f5e type=client open_count=1 channel=[id: 0xabbc540f, 0.0.0.0/0.0.0.0:54208] timeout=300.0>) 2022-02-02 04:06:44,709 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f5f type=client open_count=1 channel=[id: 0x85cd6bdd, 0.0.0.0/0.0.0.0:53643] timeout=300.0>) 2022-02-02 04:06:46,720 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f60 type=client open_count=1 channel=[id: 0x4a69f78a, 0.0.0.0/0.0.0.0:49831] timeout=300.0>) 2022-02-02 04:06:48,730 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f61 type=client open_count=1 channel=[id: 0x97c4b3ea, 0.0.0.0/0.0.0.0:44749] timeout=300.0>) 2022-02-02 04:06:50,740 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xd6f62 type=client open_count=1 channel=[id: 0xf7ad4bb7, 0.0.0.0/0.0.0.0:41890] timeout=300.0>) 2022-02-02 04:06:52,750 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xd6f63 type=client open_count=1 channel=[id: 0x036ec3f6, 0.0.0.0/0.0.0.0:43601] timeout=300.0>) 2022-02-02 04:07:00,391 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xd6f64 type=client open_count=1 channel=[id: 0xf8a45b55, 0.0.0.0/0.0.0.0:49247] timeout=300.0>) 2022-02-02 04:07:02,401 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xd6f65 type=client open_count=1 channel=[id: 0xd96fa9e0, 0.0.0.0/0.0.0.0:39667] timeout=300.0>) 2022-02-02 04:07:04,413 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xd6f66 type=client open_count=1 channel=[id: 0x42aaa82d, 0.0.0.0/0.0.0.0:42780] timeout=300.0>) 2022-02-02 04:07:06,423 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 04:07:21,420 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 04:07:34,042 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 04:07:34,042 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:07:34,062 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:07:47,148 | test | INFO | pool-5-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 0.200526381752 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 04:07:47,171 | test | INFO | pool-5-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 04:08:02,210 | test | INFO | pool-5-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 04:08:02,223 | test | INFO | pool-5-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0750000477 sec 2022-02-02 04:08:02,339 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.413223140496 | 11.45 GiB | 10.69 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.225479143179 | 11.45 GiB | 10.70 GiB | 24.25 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.17735470942 | 11.45 GiB | 10.67 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.288328945719 | 11.45 GiB | 10.76 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 0.791755686817 | 11.45 GiB | 10.65 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 04:08:02,339 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:08:02,339 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:08:02,505 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 04:08:02,960 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 04:08:03,273 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 10400 | 4.88 GiB | 167.22 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ 2022-02-02 04:08:03,273 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 04:08:12,403 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 04:08:12,411 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 04:37:07,164 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 734.400 | ...................******................................................................................**X | 3600.000 | | 2865.600 | ..............................................................................*****X | 2865.600 | | 0.000 | ****************************************************************************************************...........***.....*****.........**X | 920.400 | | 818.400 | .....................*****X | 818.400 | | 181.600 | ....*X | 181.600 | | 79.600 | .*****X | 79.600 | | 0.000 | ********************************************************************************************************************....**.*****X | 200.000 | | 40.800 | **X | 40.800 | | 0.000 | ******..**.******.......**.*****X | 400.000 | | 320.400 | .......**X | 320.400 | | 259.200 | ......****X | 259.200 | | 20.400 | **X | 20.400 | | 0.000 | **************************************************************************************************************************************************..********X | 100.000 | | 43.700 | ********..............**...*****X | 656.300 | | 202.000 | ....*X | 202.000 | | 79.600 | .*****X | 79.600 | | 0.000 | *****************************************************************************************************************************************************......**..*************X | 300.000 | | 61.200 | **X | 61.200 | | 0.000 | ******************************..*********....**..*****X | 300.000 | | 61.200 | *X | 61.200 | | 0.000 | *************************************************************************************************************************...........**..*****X | 500.000 | | 102.000 | .**X | 102.000 | | 0.000 | ************************************************************************************************************************..***.******..*X | 200.000 | | 179.600 | ...*************X | 179.600 | | 20.400 | **X | 20.400 | | 0.000 | ***********************************************************************..********..*******X | 179.600 | | 20.400 | *X | 20.400 | | 0.000 | *****************************************************************************************************************..**..******......*X | 384.300 | | 323.100 | .......*****X | 323.100 | | 36.100 | ********.**X | 100.000 | | 79.600 | .*****X | 79.600 | | 0.000 | *******************************************************************************************************************..*******X | 100.000 | | 20.400 | *X | 20.400 | | 0.000 | ******************************************************************************************X | 0.000 | +----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 04:37:07,243 | test | INFO | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..........******............................................*X | 300.000 | | 241.800 | ..........................................*****X | 241.800 | | 0.000 | *****************************************************************************************************.............................**............******.......*X | 269.100 | | 230.900 | ........................................*****X | 230.900 | | 69.700 | ...........**X | 69.700 | | 40.300 | ......*****X | 40.300 | | 0.000 | ****************************************************************************************************************************.......**..******....................................***.........****X | 300.000 | | 219.400 | ......................................**X | 219.400 | | 209.700 | .....................................*****X | 209.700 | | 129.100 | ......................*X | 129.100 | | 90.300 | ...............*****X | 90.300 | | 9.700 | *X | 9.700 | | 0.000 | *********************************************************************************************************************************************.*****........*****************....................................**.........*****X | 300.000 | | 138.800 | ........................*X | 138.800 | | 90.300 | ...............*****X | 90.300 | | 9.700 | **X | 9.700 | | 0.000 | **************************************************************************************************************************************************..............**....******..................................................................**................*****X | 550.000 | | 106.700 | ..................**X | 106.700 | | 0.000 | *****.*......**..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ******..**.******...........********X | 79.000 | | 9.700 | *X | 9.700 | | 0.000 | *************************************************************************************************************************.......**..******......................**.....*****X | 201.400 | | 127.900 | ......................**X | 127.900 | | 115.400 | ...................****X | 115.400 | | 60.900 | ..........**X | 60.900 | | 33.200 | .....*****X | 33.200 | | 0.000 | **********************************************************************************.......**..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *****************.******........*********X | 50.000 | | 40.300 | ......*****X | 40.300 | | 0.000 | **************************************************************************..............**.......*******X | 119.400 | | 82.400 | .............******..................................................................**...................*****X | 548.200 | | 104.900 | ..................**X | 104.900 | | 0.000 | ******..............**....*****X | 100.000 | | 19.400 | ..*X | 19.400 | | 0.000 | ********************************************************************************.......**..******..............**....********...******......**X | 200.700 | | 171.600 | ..............................****X | 171.600 | | 28.400 | ....**X | 28.400 | | 9.000 | *****X | 9.000 | | 0.000 | ************************.......***..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ************************************************************************************************.....................**......*****X | 150.000 | | 109.700 | ..................**X | 109.700 | | 100.000 | .................****X | 100.000 | | 19.400 | ..**X | 19.400 | | 0.000 | *****************************************************************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 04:37:07,454 | test | INFO | pool-5-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....******...................................................................................**............*****X | 50340.400 | | 18960.400 | ....................................**X | 18960.400 | | 12494.600 | .......................*****X | 12494.600 | | 8731.200 | ................*X | 8731.200 | | 8665.400 | ................*****X | 8665.400 | | 6957.900 | ............**X | 6957.900 | | 6503.300 | ...........******....**.*****X | 8863.800 | | 7466.200 | .............*X | 7466.200 | | 6791.600 | ............*****X | 6791.600 | | 5424.900 | .........***.******.....**.*****X | 8724.200 | | 8544.100 | ...............**.*********.*************.......***.*****X | 13429.400 | | 11760.100 | ......................*X | 11760.100 | | 11129.300 | .....................*****X | 11129.300 | | 9530.100 | .................*X | 9530.100 | | 9386.100 | .................******.**X | 9894.600 | | 9561.600 | .................****X | 9561.600 | | 9016.100 | ................***.******.********.**.*******X | 11025.300 | | 10751.000 | ....................*****X | 10751.000 | | 9867.000 | ..................*X | 9867.000 | | 9865.800 | ..................******..*******X | 10859.100 | | 10168.700 | ...................**X | 10168.700 | | 10057.600 | ..................*****X | 10057.600 | | 8544.300 | ...............*X | 8544.300 | | 8012.000 | ..............******..**.*****X | 9183.600 | | 7124.500 | .............*X | 7124.500 | | 6791.000 | ............******...**.********X | 9008.400 | | 8772.200 | ................****X | 8772.200 | | 7900.000 | ..............********X | 7924.600 | | 6745.100 | ............*X | 6745.100 | | 6397.500 | ...........******.....**..******...*X | 11187.200 | | 10574.100 | ....................*****X | 10574.100 | | 6210.200 | ...........**X | 6210.200 | | 5568.200 | ..........****X | 5568.200 | | 5451.000 | .........***.******.********.*******X | 7017.700 | | 5634.000 | ..........*X | 5634.000 | | 5260.900 | .........*****X | 5260.900 | | 4862.200 | ........********...***.******.......**.*****X | 10812.300 | | 8107.300 | ...............*X | 8107.300 | | 7564.200 | ..............*****X | 7564.200 | | 6352.900 | ...........*X | 6352.900 | | 6296.300 | ...........******.....***.*******X | 9349.200 | | 9294.100 | .................*****X | 9294.100 | | 8053.600 | ..............**.******.*X | 8671.200 | | 8563.700 | ................******.*******X | 9485.000 | | 8585.100 | ................********X | 8606.100 | | 8415.400 | ...............*X | 8415.400 | | 7670.300 | ..............*****X | 7670.300 | | 5106.100 | .........*X | 5106.100 | | 4971.600 | ........******...**.*****X | 6583.700 | | 5953.700 | ..........**X | 5953.700 | | 5881.400 | ..........******....**.*****X | 8216.800 | | 8093.200 | ...............*X | 8093.200 | | 8057.300 | ...............******......**.******....*X | 14016.200 | | 13676.200 | ..........................*****X | 13676.200 | | 8279.200 | ...............**X | 8279.200 | | 7395.900 | .............*****....***.*****X | 9835.400 | | 3518.100 | .....*X | 3518.100 | | 1842.700 | ..********.******......**.******.....***.******..*X | 9960.200 | | 9955.400 | ..................*****X | 9955.400 | | 9819.200 | ..................**.******...*X | 12004.000 | | 11471.100 | .....................*****X | 11471.100 | | 10146.900 | ...................********...***..*****X | 12977.500 | | 7779.500 | ..............*X | 7779.500 | | 5141.800 | .........*****X | 5141.800 | | 2089.700 | ...********.............**..******.***.****X | 10880.700 | | 7265.000 | .............**X | 7265.000 | | 5607.100 | ..........*****.***..******..*******X | 8286.300 | | 4164.800 | .......*X | 4164.800 | | 2424.700 | ...*********..*****.......***..*****.*..**.*****X | 9573.700 | | 4260.100 | .......*X | 4260.100 | | 2430.100 | ...******.**.******........***..****X | 8378.000 | | 4507.000 | .......**X | 4507.000 | | 2992.500 | ....******.....**.******...**.*****X | 7966.300 | | 4037.400 | .......**X | 4037.400 | | 3666.900 | ......*****.........***..*****X | 9402.700 | | 8427.600 | ...............*X | 8427.600 | | 8104.100 | ...............******...**.******...**.*****X | 12171.900 | | 6412.600 | ...........**X | 6412.600 | | 4845.100 | ........*****........***..******..**.*******X | 11400.700 | | 11259.000 | .....................******.....**.*****X | 14479.300 | | 10865.800 | ....................*X | 10865.800 | | 9186.600 | .................*****X | 9186.600 | | 8394.900 | ...............**..******.....*******X | 11677.900 | | 10484.700 | ...................**X | 10484.700 | | 10410.400 | ...................*****X | 10410.400 | | 10373.400 | ...................*X | 10373.400 | | 10312.200 | ...................******...**.*****X | 12213.300 | | 11516.600 | .....................*X | 11516.600 | | 11505.600 | .....................******.**X | 12027.900 | | 11516.600 | .....................****X | 11516.600 | | 9610.900 | ..................*********...*******X | 11475.000 | | 9735.500 | ..................*X | 9735.500 | | 9629.400 | ..................******....**.*****X | 12139.400 | | 11464.600 | .....................*******X | 11472.000 | | 9296.700 | .................**X | 9296.700 | | 8630.200 | ................******....**..******.*X | 12552.300 | | 12277.700 | .......................*****X | 12277.700 | | 11790.300 | ......................*******X | 12030.300 | | 11272.600 | .....................**X | 11272.600 | | 11043.500 | ....................*****...**X | 12348.400 | | 12241.500 | .......................*****X | 12241.500 | | 11131.100 | .....................*******X | 11321.000 | | 11236.800 | .....................*X | 11236.800 | | 11203.500 | .....................******.*******X | 11829.900 | | 10285.300 | ...................**X | 10285.300 | | 9512.800 | .................*****........***..*****X | 14140.500 | | 12605.300 | ........................********.********..*******X | 14282.200 | | 13043.300 | ........................**X | 13043.300 | | 12795.000 | ........................*****..**X | 13975.800 | | 13659.200 | ..........................*****X | 13659.200 | | 9101.300 | .................********....********....**.*****X | 13677.200 | | 12243.600 | .......................********.**X | 12858.400 | | 12234.500 | .......................*****X | 12234.500 | | 11250.000 | .....................********..*X | 12364.700 | | 12266.100 | .......................*****X | 12266.100 | | 5649.100 | ..........*X | 5649.100 | | 3767.600 | ......******.***..*****.......***.*****X | 9508.500 | | 9015.800 | ................********......**.******.*******X | 12999.000 | | 11463.400 | .....................**X | 11463.400 | | 11153.400 | .....................*******X | 11378.100 | | 11319.100 | .....................******...**.*****X | 13401.800 | | 9532.700 | .................*X | 9532.700 | | 8358.400 | ...............******......**..******...*******X | 13951.000 | | 13170.500 | .........................**X | 13170.500 | | 12792.800 | ........................****X | 12792.800 | | 12269.900 | .......................********X | 12480.900 | | 11765.500 | ......................*X | 11765.500 | | 11582.700 | ......................*****X | 11582.700 | | 10984.100 | ....................*X | 10984.100 | | 10336.200 | ...................*****X | 10336.200 | | 5024.300 | ........*X | 5024.300 | | 3366.600 | .....******....***..*******X | 6504.200 | | 6208.500 | ...........*****X | 6208.500 | | 3405.100 | .....*X | 3405.100 | | 2698.600 | ....******......**....******....*******X | 9723.700 | | 7353.900 | .............**X | 7353.900 | | 6788.900 | ............******.*X | 7060.600 | | 6267.200 | ...........********...******.....**.******.....**.*****X | 13989.500 | | 13842.300 | ..........................**X | 13842.300 | | 13734.500 | ..........................****X | 13734.500 | | 13288.300 | .........................********X | 13326.800 | | 11284.900 | .....................*X | 11284.900 | | 10994.100 | ....................******.....**.******.*X | 14169.200 | | 14097.700 | ...........................*****X | 14097.700 | | 10863.200 | ....................**X | 10863.200 | | 9920.300 | ..................*****.***.*****.**X | 11112.800 | | 10698.300 | ....................*****X | 10698.300 | | 10621.100 | ....................********.*******X | 11548.100 | | 10207.300 | ...................*X | 10207.300 | | 9229.300 | .................******....***.****X | 11980.700 | | 9490.100 | .................**X | 9490.100 | | 8275.100 | ...............*****X | 8275.100 | | 4523.200 | .......*X | 4523.200 | | 4446.300 | .......******.........**...******....**.******...*X | 14313.400 | | 13865.000 | ..........................X | 13865.000 | +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 04:37:08,750 | test | INFO | pool-5-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 13865.000 | ...................................................................................................X | 13865.000 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 04:37:08,868 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 31.4168902517 | 11.45 GiB | 9.06 GiB | 52.50 MiB / 3.50 GiB | 3254220 / 6522650 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 30.6049366927 | 11.45 GiB | 9.10 GiB | 24.25 MiB / 3.50 GiB | 3283807 / 6535928 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 23.6036725325 | 11.45 GiB | 9.06 GiB | 28.14 MiB / 3.50 GiB | 3260760 / 6473436 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 30.5186509964 | 11.45 GiB | 9.03 GiB | 0.0 Byte / 3.50 GiB | 3207781 / 6449977 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 32.4878298745 | 11.45 GiB | 8.98 GiB | 0.0 Byte / 3.50 GiB | 3263991 / 6557741 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2022-02-02 04:37:36,005 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 04:37:50,905 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 04:37:50,905 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 04:38:02,048 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 04:38:02,049 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 04:38:13,371 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 04:38:13,371 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 04:38:31,859 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+--------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+--------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 100000 | 9.77 GiB | 313.43 MiB | 371.21 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 527.58 MiB | 568.50 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 16287500 | 2.50 GiB | 1.68 GiB | 23.87 GiB | 25.691561013 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+--------------+ 2022-02-02 04:38:31,859 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 04:38:36,513 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during hard_failover_rebalance_out 2022-02-02 04:38:36,515 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : hard_failover_rebalance_out 2022-02-02 04:38:36,515 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:623] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 04:38:45,711 | test | INFO | pool-5-thread-1 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03999996185 seconds to complete 2022-02-02 04:38:45,713 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 8.03999996185 seconds. Reason: Wait after rebalance complete 2022-02-02 04:43:53,756 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 04:43:53,756 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 04:43:53,763 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 05:23:48,845 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.............................................................................**.......................*****X | 212.000 | | 47.400 | .....................**X | 47.400 | | 0.000 | *******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************.......................***.......*****X | 64.000 | | 14.000 | .....*X | 14.000 | | 0.000 | *******************************.....**..*****X | 16.000 | | 3.500 | *X | 3.500 | | 0.000 | *******************************************************************........******.............................*X | 80.000 | | 62.500 | ............................*****X | 62.500 | | 0.000 | *********.....**..*************X | 16.000 | | 3.500 | **X | 3.500 | | 0.000 | ********...******.......................*******X | 57.000 | | 7.000 | ..**X | 7.000 | | 0.000 | ********.******......*X | 16.000 | | 12.500 | ....*****X | 12.500 | | 0.000 | *********...........**....*****X | 32.000 | | 7.000 | ..*X | 7.000 | | 0.000 | **************.....***..******..................**.....*****X | 64.000 | | 26.500 | ...........**X | 26.500 | | 16.000 | ......****X | 16.000 | | 3.500 | *********..................**.....*****X | 51.500 | | 39.000 | .................*X | 39.000 | | 25.000 | ..........*****X | 25.000 | | 12.500 | ....***..*****X | 16.000 | | 3.500 | *X | 3.500 | | 0.000 | **********************...........***....*****X | 32.000 | | 7.000 | ..*X | 7.000 | | 0.000 | ******..............................***.........****X | 84.000 | | 31.500 | .............**X | 31.500 | | 16.000 | ......*****X | 16.000 | | 3.500 | *X | 3.500 | | 0.000 | ******.....***..****X | 16.000 | | 3.500 | **X | 3.500 | | 0.000 | ***************************************.................***.......*****X | 51.500 | | 39.000 | .................********......*X | 51.500 | | 37.500 | ................*****X | 37.500 | | 0.000 | *****************...........**........******.**X | 43.000 | | 31.000 | .............******...........**...*****X | 60.500 | | 23.000 | .........**..******...................................**..*****X | 105.000 | | 92.500 | ..........................................********X | 92.500 | | 17.500 | .......*X | 17.500 | | 3.500 | ******........................**.....*****X | 64.000 | | 26.500 | ...........***..*****........................................................................***.....*****X | 194.500 | | 42.000 | ..................*X | 42.000 | | 19.500 | ........******...........********............***.....****X | 80.000 | | 30.000 | .............**X | 30.000 | | 16.000 | ......******.........................................**............*****X | 128.000 | | 53.000 | ........................**X | 53.000 | | 32.000 | ..............****X | 32.000 | | 7.000 | ..**X | 7.000 | | 3.500 | ******..................**.....******......*X | 64.000 | | 60.500 | ...........................******.........................***.......****X | 128.500 | | 63.500 | ............................**X | 63.500 | | 56.500 | .........................******.........................................................**.........*****X | 195.500 | | 38.000 | ................*X | 38.000 | | 0.000 | *********...******......................................................**.........*****X | 141.000 | | 26.000 | ...........*X | 26.000 | | 0.000 | ******....................***........*****....................***..*****X | 108.100 | | 47.200 | .....................*X | 47.200 | | 32.000 | ..............*****X | 32.000 | | 7.000 | ..**.....******...............................**X | 84.000 | | 65.000 | .............................********.........*****X | 84.000 | | 19.000 | .......*X | 19.000 | | 3.500 | ******......*X | 16.000 | | 12.500 | ....******......................................***.............*****X | 120.000 | | 40.400 | ..................*X | 40.400 | | 16.000 | ......*****X | 16.000 | | 3.500 | **.******.................................**.......*****X | 90.500 | | 33.400 | ..............*X | 33.400 | | 12.500 | ....*****X | 12.500 | | 0.000 | ***.*****...................***..*****X | 48.500 | | 8.500 | ...*X | 8.500 | | 0.000 | ******...............................**..........*****X | 88.000 | | 33.000 | ..............**X | 33.000 | | 16.000 | ......*****X | 16.000 | | 3.500 | *X | 3.500 | | 0.000 | ******...........................**...........*****X | 81.000 | | 38.500 | .................**X | 38.500 | | 15.000 | ......*****............***.......****X | 56.000 | | 26.400 | ...........**X | 26.400 | | 24.500 | ..........******.......*X | 39.500 | | 27.500 | ...........*****X | 27.500 | | 0.000 | ********..................***..........*******X | 60.900 | | 46.600 | ....................*****X | 46.600 | | 19.500 | ........*X | 19.500 | | 12.500 | ....******..........................**..........*****X | 88.000 | | 63.000 | ............................**X | 63.000 | | 56.000 | .........................*****X | 56.000 | | 26.000 | ...........*X | 26.000 | | 19.500 | ........******...................**........*****X | 77.300 | | 76.500 | ...................................*X | 76.500 | | 55.200 | .........................*****X | 55.200 | | 46.000 | ....................**.....******..............**.....*****X | 96.000 | | 23.500 | ..........**X | 23.500 | | 0.000 | ********........******........................................*X | 102.000 | | 94.600 | ...........................................*****X | 94.600 | | 40.000 | .................**....******.............................................**........*****X | 161.000 | | 33.900 | ..............**X | 33.900 | | 13.900 | .....*****....................**X | 56.000 | | 42.100 | ..................********.......*****X | 56.000 | | 13.900 | .....*X | 13.900 | | 0.000 | ********...******............**X | 32.000 | | 24.200 | ..........****X | 24.200 | | 0.000 | *********.......**..*****X | 20.000 | | 5.000 | .*X | 5.000 | | 0.000 | ******...........................................**.............*****X | 120.000 | | 28.700 | ............**X | 28.700 | | 0.000 | **************.....**..*****X | 16.000 | | 3.900 | **X | 3.900 | | 0.000 | **************.....................**.......*****X | 60.000 | | 30.000 | .............*X | 30.000 | | 20.000 | ........******.............***....*****X | 56.000 | | 28.900 | ............*X | 28.900 | | 20.000 | ........******.......**.........******...............**X | 85.000 | | 65.000 | .............................****X | 65.000 | | 29.200 | ............***........******................*X | 78.800 | | 62.100 | ............................*****X | 62.100 | | 5.000 | .*X | 5.000 | | 0.000 | ******.....**..******.......***..*****X | 36.000 | | 33.100 | ..............*X | 33.100 | | 32.000 | ..............*****X | 32.000 | | 7.800 | ..*X | 7.800 | | 0.000 | ******...................**.......*****X | 56.000 | | 13.900 | .....**X | 13.900 | | 0.000 | ******...................**.........******......*X | 73.100 | | 67.000 | ..............................*****X | 67.000 | | 37.000 | ................**.....******.........**X | 67.800 | | 61.700 | ............................******...........................*******X | 119.300 | | 28.800 | ............*X | 28.800 | | 16.000 | ......******......***....*****.....*.........*******X | 67.000 | | 12.800 | .....*X | 12.800 | | 0.000 | ******.......**........******...............................................***............****X | 157.000 | | 108.600 | ..................................................**X | 108.600 | | 70.200 | ................................*****X | 70.200 | | 3.900 | *X | 3.900 | | 0.000 | ******......................................................**.................*****X | 152.000 | | 49.400 | ......................**X | 49.400 | | 31.600 | .............******..................*X | 67.900 | | 48.400 | .....................******................................**.................*****X | 152.000 | | 37.300 | ................**X | 37.300 | | 0.000 | *****............................***.........*****X | 80.000 | | 67.900 | ...............................*******X | 67.900 | | 31.600 | .............*X | 31.600 | | 19.900 | ........******...........***..*****..............................................***...........*****X | 168.200 | | 35.100 | ...............*X | 35.100 | | 0.000 | *X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 05:23:49,941 | test | INFO | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................**................................*****X | 272.000 | | 85.000 | ..............................**X | 85.000 | | 0.000 | ********************************************************************************************************************************************************************************************************************************************************..**X | 8.000 | | 5.500 | .*****X | 5.500 | | 0.000 | *******************************************************************************************..********X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | *********************************************************************************..*******X | 8.000 | | 2.500 | **X | 2.500 | | 0.000 | ************************************************************************......***...*****X | 26.000 | | 12.600 | ...*X | 12.600 | | 8.000 | .*****X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | ******..********X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | **************..*******X | 8.000 | | 2.500 | **X | 2.500 | | 0.000 | *********************************************************************************..**.******......**.*****X | 29.500 | | 7.500 | .**X | 7.500 | | 0.000 | ******......................**............*****X | 93.000 | | 54.500 | ...................**X | 54.500 | | 27.000 | ........****X | 27.000 | | 5.000 | **X | 5.000 | | 0.000 | **************..*X | 8.000 | | 5.500 | .*****X | 5.500 | | 0.000 | *********......**...*****X | 26.500 | | 21.000 | ......*X | 21.000 | | 13.500 | ...*****X | 13.500 | | 2.500 | **X | 2.500 | | 0.000 | *****..***.******......**.*****X | 29.500 | | 13.000 | ...*X | 13.000 | | 8.000 | .*****X | 8.000 | | 2.500 | *********...........**....*******X | 42.500 | | 37.500 | ............*****X | 37.500 | | 15.500 | ....**X | 15.500 | | 5.500 | .******....**...*****X | 26.000 | | 18.100 | .....*X | 18.100 | | 16.000 | ....******......**..*****X | 36.000 | | 20.200 | ......**X | 20.200 | | 16.000 | ....*****X | 16.000 | | 5.000 | *X | 5.000 | | 0.000 | ******......**...*****X | 26.500 | | 21.000 | ......**X | 21.000 | | 13.500 | ...*****X | 13.500 | | 8.000 | .**..******..*X | 18.500 | | 11.000 | ...*****X | 11.000 | | 0.000 | *********..***************X | 8.000 | | 2.500 | **X | 2.500 | | 0.000 | **************......**...*****X | 26.000 | | 12.600 | ...**X | 12.600 | | 8.000 | .*****X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | ******......**..*****X | 24.000 | | 13.000 | ...**X | 13.000 | | 8.000 | .*****...***.*****X | 18.000 | | 4.600 | *X | 4.600 | | 0.000 | ******......**..*****X | 24.000 | | 7.500 | .**X | 7.500 | | 0.000 | ******..............**.........******......*X | 80.100 | | 72.500 | .........................*****X | 72.500 | | 17.400 | .....**X | 17.400 | | 2.500 | *****..*...**.*****X | 18.000 | | 10.100 | ..*X | 10.100 | | 5.500 | .******....**..******....***..*****X | 38.500 | | 28.200 | .........*X | 28.200 | | 21.500 | ......*****X | 21.500 | | 18.400 | .....*X | 18.400 | | 18.000 | .....*****X | 18.000 | | 10.100 | ..*X | 10.100 | | 8.000 | .*****X | 8.000 | | 2.500 | **X | 2.500 | | 0.000 | **************........**.....*****X | 37.000 | | 26.000 | ........**X | 26.000 | | 18.500 | .....*****.........***..*****X | 47.400 | | 45.100 | ...............*X | 45.100 | | 43.100 | ..............******..............**.......*****X | 100.000 | | 85.100 | ..............................**X | 85.100 | | 74.000 | ..........................*****X | 74.000 | | 18.000 | .....*X | 18.000 | | 0.000 | **************......***...****X | 26.000 | | 7.100 | .**X | 7.100 | | 2.500 | ******......................**......*****X | 77.500 | | 23.900 | .......*X | 23.900 | | 8.000 | .********X | 8.100 | | 8.000 | .*****..***........******................*X | 76.900 | | 55.700 | ...................*****X | 55.700 | | 44.700 | ...............**.....*****X | 58.500 | | 37.200 | ............**X | 37.200 | | 23.400 | .......*****X | 23.400 | | 2.100 | ********..........**..*****X | 33.500 | | 6.600 | .**X | 6.600 | | 0.000 | **************............**...*****X | 41.900 | | 9.500 | ..*X | 9.500 | | 0.000 | ******.....**..*****X | 20.000 | | 9.700 | ..**X | 9.700 | | 8.000 | .****X | 8.000 | | 2.500 | **X | 2.500 | | 0.000 | ******..**.******...*X | 18.300 | | 17.900 | .....********X | 18.000 | | 17.700 | .....******.*X | 20.000 | | 17.900 | .....*****X | 17.900 | | 2.100 | *X | 2.100 | | 0.000 | ******...........**....*****X | 42.000 | | 17.400 | .....**X | 17.400 | | 14.900 | ....*****........**X | 36.400 | | 29.100 | .........*****X | 29.100 | | 2.100 | ********...********......***.*****X | 27.900 | | 4.200 | *X | 4.200 | | 0.000 | ******....**..******............***.****X | 53.500 | | 24.200 | .......***.******.....*X | 40.600 | | 27.000 | ........*****X | 27.000 | | 19.100 | ......**..*****X | 26.000 | | 12.500 | ...*X | 12.500 | | 10.400 | ..******...........***...****X | 47.600 | | 17.300 | .....**X | 17.300 | | 8.000 | .**************.....**..*****X | 26.000 | | 23.700 | .......**....******..................*X | 82.500 | | 77.300 | ...........................*****X | 77.300 | | 18.700 | .....**X | 18.700 | | 10.000 | ..****X | 10.000 | | 2.100 | **X | 2.100 | | 0.000 | ******..**.*****X | 10.000 | | 2.100 | ********........**...******...........***..****X | 66.200 | | 24.300 | .......**X | 24.300 | | 13.600 | ....*****X | 13.600 | | 2.500 | *X | 2.500 | | 0.000 | **************........**.*****X | 25.900 | | 4.500 | **X | 4.500 | | 0.000 | **********************.......***.****X | 23.600 | | 4.500 | **X | 4.500 | | 0.000 | ******..**.*****X | 10.000 | | 2.100 | *X | 2.100 | | 0.000 | ******.....***..*****X | 20.000 | | 4.200 | *X | 4.200 | | 0.000 | ******.........**.....*****X | 38.100 | | 19.000 | .....*X | 19.000 | | 7.900 | .*****X | 7.900 | | 0.000 | *********...*******X | 10.300 | | 8.000 | .*X | 8.000 | | 5.600 | .**************.***.*****X | 12.400 | | 10.100 | ..*X | 10.100 | | 7.700 | .******...**..******......*X | 36.200 | | 33.500 | ...........*****X | 33.500 | | 4.200 | **X | 4.200 | | 0.000 | *****..********X | 8.000 | | 2.400 | **.******........*X | 25.900 | | 23.500 | .......******...**.....*****X | 44.300 | | 27.700 | .........**X | 27.700 | | 15.700 | ....****X | 15.700 | | 2.100 | **X | 2.100 | | 0.000 | ******......**...*****X | 26.400 | | 20.800 | ......*X | 20.800 | | 13.600 | ....*****X | 13.600 | | 2.400 | *X | 2.400 | | 0.000 | *********.******.........**.******.........**..*****X | 61.200 | | 36.500 | ............*X | 36.500 | | 34.100 | ...........******...........***...*****X | 71.900 | | 19.200 | ......*X | 19.200 | | 9.300 | ..******...........*X | 39.600 | | 32.700 | ...........*****X | 32.700 | | 8.000 | .*******X | 8.000 | | 2.400 | *********..............**....*****X | 50.000 | | 25.300 | ........*X | 25.300 | | 16.000 | ....*****X | 16.000 | | 10.400 | ..**X | 10.400 | | 8.000 | .*******X | 8.000 | | 5.600 | .******........**.....*****X | 42.400 | | 20.000 | ......*X | 20.000 | | 5.600 | .**X | 5.600 | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 05:23:49,967 | test | INFO | pool-5-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****......**..*****X | 1460.600 | | 935.500 | ....**X | 935.500 | | 817.000 | ...*****X | 817.000 | | 211.200 | *X | 211.200 | | 8.600 | *******X | 8.600 | | 4.200 | ****************X | 8.300 | | 4.000 | *******X | 4.000 | | 1.000 | *************X | 160.700 | | 11.300 | ******************X | 27.100 | | 24.300 | *****X | 24.300 | | 9.600 | **********X | 102.300 | | 102.000 | *****X | 102.000 | | 9.800 | *X | 9.800 | | 5.900 | **************X | 61.600 | | 32.100 | *X | 32.100 | | 24.700 | *************X | 103.700 | | 94.700 | **X | 94.700 | | 81.900 | *****X | 81.900 | | 44.900 | *X | 44.900 | | 44.000 | *****X | 44.000 | | 11.900 | **X | 11.900 | | 9.600 | *************X | 101.200 | | 62.600 | *X | 62.600 | | 53.700 | ******..***..******...*X | 1334.700 | | 1312.500 | ......*****X | 1312.500 | | 896.500 | ....*X | 896.500 | | 855.700 | ....******....***..******..*X | 2193.900 | | 2105.700 | ...........*****X | 2105.700 | | 1547.700 | ........*X | 1547.700 | | 1260.300 | ......*****X | 1260.300 | | 578.000 | ..***.******......**.******................***.....****X | 5414.100 | | 3987.900 | ......................**X | 3987.900 | | 3358.900 | ..................*****X | 3358.900 | | 962.000 | ....*X | 962.000 | | 366.300 | .*****X | 366.300 | | 314.500 | **X | 314.500 | | 305.500 | *****X | 305.500 | | 284.900 | *********X | 288.100 | | 270.300 | *****X | 270.300 | | 254.600 | **X | 254.600 | | 240.300 | *****X | 240.300 | | 149.200 | ********.****************X | 290.200 | | 137.200 | *X | 137.200 | | 118.800 | ******.********X | 296.100 | | 184.800 | *X | 184.800 | | 172.300 | *************X | 255.900 | | 235.600 | **X | 235.600 | | 223.700 | *****X | 223.700 | | 72.600 | *X | 72.600 | | 53.400 | ******.****************X | 293.500 | | 287.700 | *X | 287.700 | | 275.900 | *****X | 275.900 | | 165.700 | **X | 165.700 | | 154.000 | *****.********X | 276.700 | | 185.900 | *X | 185.900 | | 176.300 | *****X | 176.300 | | 120.000 | **X | 120.000 | | 104.100 | *************X | 155.700 | | 138.700 | *****************.*********X | 265.100 | | 264.800 | *****X | 264.800 | | 157.500 | **X | 157.500 | | 143.700 | ********.*************X | 296.800 | | 216.000 | **X | 216.000 | | 192.500 | *****X | 192.500 | | 109.100 | *X | 109.100 | | 86.600 | **************..***.******..*X | 899.700 | | 852.300 | ....*****X | 852.300 | | 275.600 | *X | 275.600 | | 188.400 | ********X | 207.600 | | 184.600 | **************.**.******...**.*****X | 1245.900 | | 1157.700 | .....***.******.....**.*****X | 2341.200 | | 1765.600 | .........*X | 1765.600 | | 1524.800 | ........******.***.*****X | 1858.000 | | 966.200 | ....*X | 966.200 | | 806.500 | ...******.......********.**.*****X | 2317.200 | | 1719.700 | .........**X | 1719.700 | | 1544.100 | ........******...*******X | 2108.300 | | 1150.000 | .....*X | 1150.000 | | 958.600 | ....******...***..*****........**X | 3080.300 | | 2978.200 | ................*****X | 2978.200 | | 1384.500 | .......*X | 1384.500 | | 1013.900 | .....******......**...******.**X | 2762.800 | | 2596.800 | ..............******..**.******.**.*****X | 3478.700 | | 2560.900 | ..............**X | 2560.900 | | 2323.400 | ............*****........*********......**..*****X | 5007.400 | | 2915.300 | ................*X | 2915.300 | | 2189.900 | ............*****X | 2189.900 | | 1759.800 | .........**X | 1759.800 | | 1615.000 | ........*****.............***.....*****X | 4638.800 | | 3007.400 | ................*X | 3007.400 | | 2467.000 | .............*****X | 2467.000 | | 2116.600 | ...........*X | 2116.600 | | 2106.800 | ...........******............***........*****...............**X | 7967.900 | | 7777.200 | .............................................******....**...*X | 8983.000 | | 8983.000 | ....................................................***X | 8983.000 | | 6714.500 | ......................................*X | 6714.500 | | 5958.800 | ..................................*****X | 5958.800 | | 5161.600 | .............................**X | 5161.600 | | 5142.700 | .............................**X | 5142.700 | | 5142.700 | .............................***....*X | 5767.000 | | 5743.000 | .................................*****X | 5743.000 | | 5192.000 | .............................*X | 5192.000 | | 5111.600 | .............................*****X | 5111.600 | | 4187.300 | .......................**X | 4187.300 | | 3788.700 | .....................*****.***.******.*X | 4303.200 | | 4266.800 | ........................**X | 4266.800 | | 4266.800 | ........................**X | 4266.800 | | 4038.400 | .......................********.**X | 4252.600 | | 4104.400 | .......................*******X | 4134.600 | | 3861.500 | .....................*****X | 3861.500 | | 1314.600 | ......*X | 1314.600 | | 861.200 | ....******...............**......******..........***..*****X | 6422.100 | | 2647.800 | ..............*X | 2647.800 | | 1540.400 | ........*****X | 1540.400 | | 1307.400 | ......**.******....................***.....*****X | 5634.500 | | 3660.600 | ....................*X | 3660.600 | | 3341.300 | ..................******....********.....*X | 4762.400 | | 4645.400 | ..........................*****X | 4645.400 | | 3607.000 | ....................***.*****................***...****X | 6992.500 | | 6992.500 | ........................................X | 6992.500 | | 3794.200 | .....................*X | 3794.200 | | 2909.100 | ................******.......**...*X | 4618.100 | | 4618.100 | ..........................****...............***.....*****..........***...*****X | 10219.800 | | 9777.900 | .........................................................*X | 9777.900 | | 9149.400 | .....................................................X | 9149.400 | | 9149.400 | .....................................................*****...**..*X | 9977.600 | | 9977.600 | ..........................................................***X | 9977.600 | | 8453.500 | .................................................**X | 8453.500 | | 7695.400 | ............................................****X | 7695.400 | | 5959.900 | ..................................**X | 5959.900 | | 5672.200 | ................................**X | 5672.200 | | 5672.200 | ................................**.........*.....**.....******....................**...........*****X | 13989.100 | | 12088.200 | ......................................................................*X | 12088.200 | | 10756.100 | ..............................................................******.............**...*****X | 13442.500 | | 11024.000 | ................................................................*X | 11024.000 | | 9726.600 | ........................................................*X | 9726.600 | | 9726.600 | ........................................................***X | 9726.600 | | 6930.000 | ........................................**X | 6930.000 | | 6809.200 | .......................................*****.....X | 7580.200 | | 7580.200 | ............................................**..X | 7919.800 | | 7919.800 | ..............................................*X | 7919.800 | | 7919.800 | ..............................................***.....**..*****X | 9177.400 | | 7499.600 | ...........................................*X | 7499.600 | | 5940.400 | ..................................******..............X | 8392.400 | | 8392.400 | ................................................**..............*****..................................***...****X | 16807.300 | | 9420.500 | .......................................................**X | 9420.500 | | 7779.700 | .............................................**X | 7779.700 | | 7779.700 | .............................................*X | 7779.700 | | 7779.700 | .............................................*................**..*****X | 10908.600 | | 7868.000 | .............................................*X | 7868.000 | | 7572.500 | ............................................X | 7572.500 | | 7572.500 | ............................................**X | 7572.500 | | 7572.500 | ............................................*X | 7572.500 | | 7110.100 | .........................................**X | 7110.100 | | 6385.000 | ....................................****X | 6385.000 | | 4479.300 | .........................*X | 4479.300 | | 4479.300 | .........................X | 4479.300 | | 4240.800 | ........................****X | 4240.800 | | 4240.800 | ........................*.........................**...........******.*X | 10518.600 | | 9656.000 | ........................................................*****X | 9656.000 | | 8198.100 | ...............................................*X | 8198.100 | | 7643.400 | ............................................*****X | 7643.400 | | 7178.900 | .........................................***...*****...***....******.........................**.......*****X | 14169.400 | | 13206.400 | .............................................................................*X | 13206.400 | | 11860.800 | .....................................................................*****X | 11860.800 | | 7198.800 | .........................................*X | 7198.800 | | 6466.500 | .....................................******...........***..****X | 8726.900 | | 8045.100 | ..............................................**X | 8045.100 | | 7804.300 | .............................................*****X | 7804.300 | | 7066.200 | .........................................********.....**.**X | 8113.800 | | 8113.800 | ...............................................**X | 8113.800 | | 5895.200 | ..................................*X | 5895.200 | | 5359.700 | ..............................*****X | 5359.700 | | 5321.600 | ..............................*X | 5321.600 | | 5009.800 | ............................******.................***.........*****..............***...****X | 12106.200 | | 8696.000 | ..................................................**X | 8696.000 | | 7195.000 | .........................................*****X | 7195.000 | | 7070.000 | .........................................**....******..X | 8186.200 | | 8186.200 | ...............................................X | 8186.200 | | 7845.800 | .............................................******..**X | 8152.600 | | 8005.300 | ..............................................****X | 8005.300 | | 7943.600 | ..............................................***..*****X | 8351.700 | | 7294.000 | ..........................................*X | 7294.000 | | 6945.000 | ........................................******.*X | 7065.500 | | 6444.100 | .....................................*****X | 6444.100 | | 4982.900 | ............................*******X | 5029.900 | | 4021.300 | ......................**X | 4021.300 | | 3291.700 | ..................********......******.............**..*****X | 6806.100 | | 6789.100 | .......................................*X | 6789.100 | | 6305.100 | ....................................*****X | 6305.100 | | 6283.900 | ....................................**..******......***....*****..**X | 8687.500 | | 8453.000 | .................................................*****X | 8453.000 | | 6887.600 | .......................................*X | 6887.600 | | 6560.300 | ......................................******...............**.....*****X | 10059.000 | | 8073.000 | ...............................................*X | 8073.000 | | 7481.900 | ...........................................******.*X | 7686.000 | | 7330.800 | ..........................................*****X | 7330.800 | | 7082.900 | .........................................**..*****X | 7526.700 | | 6674.100 | ......................................**X | 6674.100 | | 5994.000 | ..................................*****.***...*****X | 6697.900 | | 6447.600 | .....................................*X | 6447.600 | | 6369.800 | ....................................******......**..******...........*X | 9577.600 | | 9443.700 | .......................................................*****X | 9443.700 | | 8161.100 | ...............................................**...*****X | 8632.700 | | 7128.400 | .........................................*X | 7128.400 | | 6337.800 | ....................................*****X | 6337.800 | | 5659.600 | ................................**X | 5659.600 | | 5144.200 | .............................******.......**.....******...........**....*****X | 9583.200 | | 6911.700 | ........................................*X | 6911.700 | | 4999.500 | ............................*****X | 4999.500 | | 4225.800 | ........................***....******.*X | 5161.400 | | 4725.100 | ...........................*****X | 4725.100 | | 3625.200 | ....................********.....*******X | 4453.200 | | 3433.700 | ...................*X | 3433.700 | | 2827.600 | ...............*****X | 2827.600 | | 1998.300 | ..........***.****X | 2025.200 | | 1678.300 | ........**X | 1678.300 | | 1618.500 | ........******.......**..*****X | 3136.700 | | 2245.900 | ............*X | 2245.900 | | 1973.100 | ..........******.....***.*****X | 2993.800 | | 1963.300 | ..........*X | 1963.300 | | 1582.300 | ........******.....**..*****X | 2842.600 | | 2786.700 | ...............**X | 2786.700 | | 2705.000 | ...............*****......***....******........*X | 5714.800 | | 5469.500 | ...............................******.........**.......*******X | 8138.200 | | 7626.200 | ............................................*****X | 7626.200 | | 7343.200 | ..........................................*******X | 7381.300 | | 7356.100 | ..........................................***.******.....*X | 8389.200 | | 8268.900 | ................................................*****X | 8268.900 | | 5089.400 | .............................*X | 5089.400 | | 4653.800 | ..........................******...................**.....*****X | 8627.800 | | 8457.800 | .................................................**X | 8457.800 | | 8203.800 | ...............................................****X | 8203.800 | | 5954.100 | ..................................**X | 5954.100 | | 5407.300 | ...............................******........**..*****X | 7195.300 | | 5437.700 | ...............................*X | 5437.700 | | 4758.500 | ...........................******.........**.....******.......**...******......*X | 9855.300 | | 9582.700 | ........................................................*****X | 9582.700 | | 7576.100 | ............................................**X | 7576.100 | | 6868.500 | .......................................*****....***.....*****X | 8249.900 | | 8159.500 | ...............................................*X | 8159.500 | | 7572.100 | ............................................*****X | 7572.100 | | 7136.900 | .........................................**.******......**....******........***.****X | 10467.700 | | 9071.400 | ....................................................**X | 9071.400 | | 8642.400 | ..................................................******....**..******.**.*****X | 9947.600 | | 8660.200 | ..................................................*X | 8660.200 | | 7791.700 | .............................................*****X | 7791.700 | | 7702.400 | ............................................***....*****...............***...*****X | 11307.000 | | 10373.500 | ............................................................*X | 10373.500 | | 9994.900 | ..........................................................******..........**..*****X | 12009.900 | | 11232.700 | .................................................................*******X | 11247.000 | | 10063.800 | ..........................................................**X | 10063.800 | | 9154.400 | .....................................................****X | 9154.400 | | 8569.200 | .................................................X | 8569.200 | | 8259.200 | ................................................**...******.........................**.....*****X | 13812.000 | | 10953.300 | ................................................................*X | 10953.300 | | 10017.600 | ..........................................................**X | 10017.600 | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 05:23:51,244 | test | INFO | pool-5-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------+-------+ | 5.600 | ...................................................................................................X | 5.600 | +-------+------------------------------------------------------------------------------------------------------+-------+ 2022-02-02 05:23:51,265 | test | INFO | pool-5-thread-22 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 10017.600 | ...................................................................................................X | 10017.600 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 05:23:51,415 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 05:23:51,437 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 05:24:27,963 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 05:24:27,973 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 05:24:28,075 | test | INFO | pool-5-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 4.54089312594 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 9.81355505165 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 5.24335173106 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.37589274527 | --- OUT ---> | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 8.23455233291 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 05:24:28,096 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 05:24:38,134 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 24.04 2022-02-02 05:24:48,170 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 28.39 2022-02-02 05:24:58,206 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 32.36 2022-02-02 05:25:08,246 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 35.5 2022-02-02 05:25:18,289 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 36.58 2022-02-02 05:25:28,328 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 39.44 2022-02-02 05:25:38,362 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 44.17 2022-02-02 05:25:48,407 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 45.09 2022-02-02 05:25:58,447 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 49.76 2022-02-02 05:26:08,480 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 52.82 2022-02-02 05:26:18,523 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 53.04 2022-02-02 05:26:28,565 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 54.73 2022-02-02 05:26:38,605 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 56.95 2022-02-02 05:26:48,647 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 58.95 2022-02-02 05:26:58,680 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 61.82 2022-02-02 05:27:08,734 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 63.46 2022-02-02 05:27:18,809 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 63.62 2022-02-02 05:27:28,848 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 63.62 2022-02-02 05:27:38,887 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 63.67 2022-02-02 05:27:48,930 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 63.67 2022-02-02 05:27:58,969 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 66.56 2022-02-02 05:28:09,020 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 66.72 2022-02-02 05:28:19,062 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 66.83 2022-02-02 05:28:29,101 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 67.26 2022-02-02 05:28:39,134 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 67.48 2022-02-02 05:28:49,174 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 67.75 2022-02-02 05:28:59,233 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 67.91 2022-02-02 05:29:09,292 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 68.01 2022-02-02 05:29:19,361 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 68.12 2022-02-02 05:29:29,400 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 68.28 2022-02-02 05:29:39,447 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 68.45 2022-02-02 05:29:49,486 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 68.61 2022-02-02 05:29:59,526 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 68.88 2022-02-02 05:30:09,592 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 69.31 2022-02-02 05:30:19,638 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 69.58 2022-02-02 05:30:29,671 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 69.85 2022-02-02 05:30:39,710 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 70.23 2022-02-02 05:30:49,746 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 70.44 2022-02-02 05:30:59,792 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 70.82 2022-02-02 05:31:09,839 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 70.98 2022-02-02 05:31:19,884 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 71.36 2022-02-02 05:31:29,926 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 71.63 2022-02-02 05:31:39,966 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 71.68 2022-02-02 05:31:50,005 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 71.85 2022-02-02 05:32:00,040 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 72.06 2022-02-02 05:32:10,078 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 72.17 2022-02-02 05:32:20,114 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 72.23 2022-02-02 05:32:30,174 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 72.45 2022-02-02 05:32:40,243 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 72.72 2022-02-02 05:32:50,289 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 72.83 2022-02-02 05:33:00,335 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 73.1 2022-02-02 05:33:10,387 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 73.26 2022-02-02 05:33:20,424 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 73.37 2022-02-02 05:33:30,467 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 73.64 2022-02-02 05:33:40,509 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 74.13 2022-02-02 05:33:50,555 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 74.24 2022-02-02 05:34:00,601 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 74.62 2022-02-02 05:34:10,641 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 74.94 2022-02-02 05:34:20,683 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 74.94 2022-02-02 05:34:30,723 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 75.0 2022-02-02 05:34:40,767 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 75.16 2022-02-02 05:34:50,811 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 75.32 2022-02-02 05:35:00,855 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 75.49 2022-02-02 05:35:10,888 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 75.76 2022-02-02 05:35:20,927 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 76.08 2022-02-02 05:35:30,967 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 76.24 2022-02-02 05:35:41,003 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 76.46 2022-02-02 05:35:51,039 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 76.67 2022-02-02 05:36:01,078 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 77.05 2022-02-02 05:36:11,119 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 77.37 2022-02-02 05:36:21,155 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 77.59 2022-02-02 05:36:31,194 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 77.86 2022-02-02 05:36:41,247 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 78.08 2022-02-02 05:36:51,285 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 78.24 2022-02-02 05:37:01,322 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 78.46 2022-02-02 05:37:11,358 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 78.68 2022-02-02 05:37:21,392 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 78.95 2022-02-02 05:37:31,434 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.06 2022-02-02 05:37:41,509 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.22 2022-02-02 05:37:51,572 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.28 2022-02-02 05:38:01,611 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.44 2022-02-02 05:38:11,648 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.6 2022-02-02 05:38:21,690 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.65 2022-02-02 05:38:31,734 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 79.87 2022-02-02 05:38:41,783 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 80.03 2022-02-02 05:38:51,832 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 80.19 2022-02-02 05:39:01,868 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 80.46 2022-02-02 05:39:11,913 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 80.63 2022-02-02 05:39:21,973 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 80.79 2022-02-02 05:39:32,020 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 81.01 2022-02-02 05:39:42,072 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 81.17 2022-02-02 05:39:52,111 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 81.61 2022-02-02 05:40:02,176 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 81.77 2022-02-02 05:40:12,256 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 81.99 2022-02-02 05:40:22,295 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 82.26 2022-02-02 05:40:32,335 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 82.48 2022-02-02 05:40:42,388 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 82.69 2022-02-02 05:40:52,427 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 83.07 2022-02-02 05:41:02,463 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 83.44 2022-02-02 05:41:12,500 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 83.93 2022-02-02 05:41:22,539 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 84.2 2022-02-02 05:41:32,578 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 84.47 2022-02-02 05:41:42,615 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 84.8 2022-02-02 05:41:52,650 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 85.23 2022-02-02 05:42:02,684 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 85.4 2022-02-02 05:42:12,716 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 85.67 2022-02-02 05:42:22,760 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 85.94 2022-02-02 05:42:32,803 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 86.11 2022-02-02 05:42:42,844 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 86.38 2022-02-02 05:42:52,884 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 86.65 2022-02-02 05:43:02,938 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 86.7 2022-02-02 05:43:12,980 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 87.02 2022-02-02 05:43:23,046 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 87.19 2022-02-02 05:43:33,082 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 87.3 2022-02-02 05:43:43,148 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 87.51 2022-02-02 05:43:53,184 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 87.78 2022-02-02 05:44:03,236 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 87.89 2022-02-02 05:44:13,276 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 88.38 2022-02-02 05:44:23,315 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 88.76 2022-02-02 05:44:33,355 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 89.03 2022-02-02 05:44:43,456 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 89.3 2022-02-02 05:44:53,512 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 89.62 2022-02-02 05:45:03,598 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 90.27 2022-02-02 05:45:13,650 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 90.59 2022-02-02 05:45:23,686 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 90.98 2022-02-02 05:45:33,726 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 91.19 2022-02-02 05:45:43,765 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 91.57 2022-02-02 05:45:53,799 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 91.79 2022-02-02 05:46:03,835 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 91.95 2022-02-02 05:46:13,924 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.01 2022-02-02 05:46:24,039 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.01 2022-02-02 05:46:34,130 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.11 2022-02-02 05:46:44,180 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.22 2022-02-02 05:46:54,223 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.33 2022-02-02 05:47:04,269 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.65 2022-02-02 05:47:14,305 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 92.87 2022-02-02 05:47:24,344 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 93.2 2022-02-02 05:47:34,382 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 93.31 2022-02-02 05:47:44,420 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 93.69 2022-02-02 05:47:54,459 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 93.91 2022-02-02 05:48:04,500 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.02 2022-02-02 05:48:14,539 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.13 2022-02-02 05:48:24,582 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.18 2022-02-02 05:48:34,632 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.29 2022-02-02 05:48:44,673 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.34 2022-02-02 05:48:54,711 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.4 2022-02-02 05:49:04,744 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.62 2022-02-02 05:49:14,785 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.78 2022-02-02 05:49:24,821 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 94.94 2022-02-02 05:49:34,858 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 95.16 2022-02-02 05:49:44,895 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 95.32 2022-02-02 05:49:54,934 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 95.7 2022-02-02 05:50:04,967 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 95.81 2022-02-02 05:50:15,005 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 95.81 2022-02-02 05:50:25,042 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 95.87 2022-02-02 05:50:35,075 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 96.03 2022-02-02 05:50:38,398 | test | INFO | pool-5-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.........................******...........................................................................**X | 1437.900 | | 1075.200 | .........................................................................*****X | 1075.200 | | 3.800 | *X | 3.800 | | 0.000 | ******************************************.**************.**..******...**X | 109.800 | | 102.200 | ......*****.......***....*****X | 260.300 | | 154.400 | .........*X | 154.400 | | 118.700 | .......*****X | 118.700 | | 116.200 | .......**X | 116.200 | | 103.400 | ......*****X | 103.400 | | 31.700 | .*X | 31.700 | | 15.300 | ******.*******X | 34.500 | | 8.900 | *X | 8.900 | | 0.000 | *****************.******...**..******.*X | 103.300 | | 76.600 | ....*****X | 76.600 | | 42.100 | .***.******....*X | 103.200 | | 95.700 | .....******.**.*****X | 118.800 | | 44.800 | ..**X | 44.800 | | 23.000 | *****.*********..*X | 65.100 | | 56.200 | ..******...**.*****X | 111.100 | | 25.700 | **X | 25.700 | | 0.000 | **************.......**...*******X | 156.900 | | 136.600 | ........*****X | 136.600 | | 88.200 | .....X | 88.200 | | 88.200 | .....*X | 88.200 | | 88.100 | .....*****X | 88.100 | | 62.500 | ...*X | 62.500 | | 46.000 | ..*****X | 46.000 | | 8.900 | *X | 8.900 | | 0.000 | ******...***.*****X | 65.100 | | 16.500 | *X | 16.500 | | 0.000 | ******...**.*****X | 69.000 | | 17.800 | **X | 17.800 | | 0.000 | ****************.******.**X | 30.600 | | 26.800 | *****.***.*****X | 45.900 | | 22.900 | *X | 22.900 | | 15.300 | **************......********X | 108.600 | | 59.900 | ...*X | 59.900 | | 53.700 | ..*****X | 53.700 | | 14.100 | *X | 14.100 | | 0.000 | *********.******....X | 75.400 | | 75.400 | ....X | 75.400 | | 75.300 | ....*X | 75.300 | | 75.300 | ....***X | 75.300 | | 26.800 | *X | 26.800 | | 19.200 | *****X | 19.200 | | 5.100 | **X | 5.100 | | 0.000 | **************..**..******.**X | 72.700 | | 60.100 | ...****X | 60.100 | | 9.000 | **X | 9.000 | | 0.000 | ********.******..**.*****X | 61.200 | | 49.700 | ..**X | 49.700 | | 45.900 | ..*****X | 45.900 | | 11.600 | ********....*X | 61.300 | | 52.400 | ..*****X | 52.400 | | 15.200 | **********X | 19.200 | | 11.500 | ******....**..*****X | 88.200 | | 48.600 | ..********..**X | 85.400 | | 63.700 | ...*****X | 63.700 | | 15.300 | ********.**.******..**X | 77.900 | | 75.300 | ....*****...***..*****X | 154.600 | | 80.600 | ....*X | 80.600 | | 48.600 | ..*****X | 48.600 | | 37.100 | .**.******...*X | 95.700 | | 94.600 | .....*****X | 94.600 | | 18.000 | **X | 18.000 | | 0.000 | ******.**.*****X | 30.600 | | 7.600 | *X | 7.600 | | 0.000 | *********.*****X | 15.300 | | 3.800 | *X | 3.800 | | 0.000 | ***********************..**.*****X | 53.700 | | 14.000 | *X | 14.000 | | 3.900 | ******..*******X | 30.600 | | 5.100 | ***.******....*******X | 76.500 | | 51.000 | ..*X | 51.000 | | 34.300 | .*****X | 34.300 | | 0.000 | ****************..**X | 30.600 | | 29.400 | .****X | 29.400 | | 18.000 | *********....**..*****X | 107.400 | | 51.200 | ..**X | 51.200 | | 30.600 | .****X | 30.600 | | 19.100 | **X | 19.100 | | 15.300 | *****...***.*****X | 80.400 | | 32.100 | .*X | 32.100 | | 15.300 | ********.*****X | 30.600 | | 7.700 | **X | 7.700 | | 3.800 | *****.********X | 26.800 | | 3.800 | ********...*X | 48.600 | | 44.700 | ..*****X | 44.700 | | 5.100 | **X | 5.100 | | 0.000 | *****.***.*****X | 34.500 | | 20.400 | *X | 20.400 | | 15.300 | ******..********...**.*****X | 107.400 | | 42.300 | .**X | 42.300 | | 19.200 | *****X | 19.200 | | 5.100 | *X | 5.100 | | 0.000 | **************.********.*******X | 34.500 | | 9.000 | **X | 9.000 | | 0.000 | **************..**..*******X | 57.600 | | 40.800 | .******.***.****X | 71.400 | | 62.600 | ...**X | 62.600 | | 43.500 | ..****X | 43.500 | | 3.900 | *********.....**.*****X | 86.700 | | 15.600 | *X | 15.600 | | 0.000 | *********.****X | 15.300 | | 3.900 | **X | 3.900 | | 0.000 | **************.*X | 15.300 | | 11.400 | *****X | 11.400 | | 0.000 | ********..********.***.*****X | 65.100 | | 16.800 | *X | 16.800 | | 0.000 | **************.***.*****X | 30.600 | | 7.800 | *X | 7.800 | | 0.000 | *************************.*****X | 15.300 | | 3.900 | ********...*******X | 57.300 | | 20.400 | **X | 20.400 | | 15.300 | ****X | 15.300 | | 3.900 | **X | 3.900 | | 0.000 | ************************************************************************.***.****X | 34.500 | | 25.300 | X | 25.300 | | 9.000 | *X | 9.000 | | 0.000 | *************************.*******X | 19.200 | | 11.400 | ********.******************************X | 15.300 | | 3.900 | *X | 3.900 | | 0.000 | *********X | 0.000 | +----------+-----------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 05:50:38,408 | test | INFO | pool-5-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **............................******........................................................................**X | 329.000 | | 237.200 | .......................................................................*****X | 237.200 | | 2.200 | *X | 2.200 | | 0.000 | **********************.***.*****X | 7.700 | | 2.100 | *X | 2.100 | | 0.000 | ***********************......**...*****X | 32.500 | | 29.800 | ........*X | 29.800 | | 22.700 | .....******..................X | 81.200 | | 81.200 | .......................*..........*****X | 113.200 | | 56.700 | ................**X | 56.700 | | 30.700 | ........******...........**....*****X | 79.400 | | 33.400 | .........*X | 33.400 | | 20.400 | .....*******.**.*****..**X | 33.100 | | 19.600 | ....*****X | 19.600 | | 0.000 | ********.***.*****....***......******.....*X | 58.500 | | 44.400 | ............*****X | 44.400 | | 22.700 | .....*******X | 22.800 | | 18.000 | ....**X | 18.000 | | 12.400 | ..*****.********X | 15.200 | | 9.300 | .*X | 9.300 | | 7.600 | .*****X | 7.600 | | 2.800 | *X | 2.800 | | 0.000 | ******........***....*****X | 40.000 | | 21.800 | .....*X | 21.800 | | 15.200 | ...*****X | 15.200 | | 5.600 | *X | 5.600 | | 0.000 | ***************.......**.******.....**.....*****X | 60.800 | | 52.300 | ..............**X | 52.300 | | 37.200 | ..........****X | 37.200 | | 19.000 | ....***.******.......*X | 43.800 | | 39.200 | ..........*****X | 39.200 | | 18.900 | ....*X | 18.900 | | 17.200 | ....***************.*X | 19.900 | | 19.200 | ....*****X | 19.200 | | 4.200 | ********...**X | 15.200 | | 12.400 | ..*****..***.*****X | 20.000 | | 5.600 | *X | 5.600 | | 0.000 | ******..***..****X | 15.200 | | 10.400 | ..**X | 10.400 | | 7.600 | .******...**..******..**X | 30.300 | | 27.500 | .......****X | 27.500 | | 12.500 | ..**X | 12.500 | | 4.800 | *****X | 4.800 | | 0.000 | ********......**..*****X | 27.600 | | 19.100 | ....**X | 19.100 | | 15.200 | ...****X | 15.200 | | 9.300 | .***..******.*X | 18.000 | | 9.600 | .*****X | 9.600 | | 0.000 | ********.***...******.*X | 18.000 | | 9.600 | .*****X | 9.600 | | 4.800 | ***.**************....**...*****X | 29.700 | | 28.600 | .......**X | 28.600 | | 19.900 | .....****X | 19.900 | | 10.300 | ..**X | 10.300 | | 9.600 | .**************....**..******..***.******........................**.......*****X | 140.500 | | 54.800 | ...............*X | 54.800 | | 30.400 | ........******....................***........****X | 123.000 | | 89.200 | ..........................X | 89.200 | | 77.100 | ......................*X | 77.100 | | 47.800 | .............********...*****X | 56.400 | | 26.400 | .......**X | 26.400 | | 15.200 | ...*****....***...*****X | 38.000 | | 31.100 | ........*X | 31.100 | | 22.000 | .....*****X | 22.000 | | 9.700 | .**.******.**.******.**X | 20.000 | | 12.300 | ..******...**..*****X | 26.800 | | 7.000 | .*X | 7.000 | | 0.000 | ******.***.*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ******.**.*****X | 7.600 | | 1.700 | **X | 1.700 | | 0.000 | *****...........***.....*****X | 55.200 | | 16.700 | ....*X | 16.700 | | 4.200 | ******......*******X | 25.400 | | 9.300 | .***..******........*X | 42.000 | | 35.400 | .........X | 35.400 | | 35.400 | .........****X | 35.400 | | 17.200 | ....*X | 17.200 | | 12.300 | ..*****X | 12.300 | | 0.000 | ********....**.*****X | 18.900 | | 16.200 | ...*******X | 16.300 | | 10.400 | ..**X | 10.400 | | 4.800 | ******.......**...*****X | 36.400 | | 16.600 | ....**X | 16.600 | | 9.600 | .****X | 9.600 | | 6.900 | .********.**X | 10.300 | | 9.600 | .****X | 9.600 | | 2.100 | **X | 2.100 | | 0.000 | ******...**..*****X | 18.000 | | 15.800 | ...*X | 15.800 | | 12.400 | ..*****X | 12.400 | | 8.700 | .***.******..*X | 16.900 | | 12.800 | ..******..********........***..****X | 51.500 | | 35.400 | .........**X | 35.400 | | 34.400 | .........******...*******X | 46.000 | | 16.000 | ...*X | 16.000 | | 7.600 | .*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ******..***.******..*X | 17.200 | | 14.400 | ...******...**....******......*X | 58.900 | | 54.600 | ...............*****X | 54.600 | | 20.900 | .....**X | 20.900 | | 17.300 | ....****X | 17.300 | | 13.000 | ..**X | 13.000 | | 7.500 | .*****X | 7.500 | | 5.900 | **.*****X | 7.600 | | 1.700 | **.******....*X | 17.200 | | 13.400 | ...******...***....*****.....**X | 55.500 | | 53.400 | ...............****X | 53.400 | | 10.000 | ..**X | 10.000 | | 0.000 | ******..**..******...........*******X | 50.100 | | 19.000 | ....**X | 19.000 | | 15.200 | ...****X | 15.200 | | 3.400 | ***..******..........*X | 44.000 | | 32.100 | ........*****X | 32.100 | | 0.000 | *************************.........**...*****X | 42.000 | | 9.300 | .*X | 9.300 | | 1.700 | ******......***.******.**.*****X | 30.400 | | 6.800 | .*X | 6.800 | | 0.000 | ******.***.*************X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ******.**.******...........**..*****X | 49.500 | | 9.300 | .**X | 9.300 | | 1.700 | *****..**X | 7.600 | | 5.900 | *****X | 5.900 | | 0.000 | ******************************************************************........***..****X | 34.400 | | 7.600 | .**X | 7.600 | | 0.000 | **************....****************X | 15.200 | | 9.300 | .*X | 9.300 | | 7.600 | .******.....**..*****X | 30.400 | | 18.600 | ....*X | 18.600 | | 16.900 | ....******.**X | 22.800 | | 21.100 | .....*************X | 22.800 | | 5.100 | *X | 5.100 | | 0.000 | *********X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 05:50:39,459 | test | INFO | pool-5-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...**X | 638.100 | | 638.100 | ..***...............................***..X | 6052.500 | | 6052.500 | ...................................***X | 6052.500 | | 2442.600 | .............**X | 2442.600 | | 2070.700 | ...........******.........**..******.*X | 3979.000 | | 3901.500 | ......................*****X | 3901.500 | | 2451.400 | .............**X | 2451.400 | | 2348.600 | .............*****..*....**.*******X | 3525.900 | | 3408.700 | ...................******.....**..**X | 4604.700 | | 4604.700 | ..........................**X | 4604.700 | | 3848.700 | ......................***.******...............................................**..........*****X | 13479.300 | | 7197.200 | ..........................................*X | 7197.200 | | 6014.900 | ...................................******..................................***.......****X | 12687.200 | | 8574.600 | ...................................................**X | 8574.600 | | 7005.600 | .........................................*****X | 7005.600 | | 5463.100 | ................................*X | 5463.100 | | 5385.400 | ...............................*****X | 5385.400 | | 5359.900 | ...............................*X | 5359.900 | | 5009.500 | .............................*****X | 5009.500 | | 4189.700 | ........................*X | 4189.700 | | 4077.300 | .......................******...***...X | 5052.700 | | 5052.700 | .............................****.....................***.....*****X | 9356.400 | | 7415.400 | ............................................*X | 7415.400 | | 6870.500 | ........................................******.............................................**............*****X | 16201.700 | | 10036.900 | ............................................................*X | 10036.900 | | 8100.400 | ................................................******......***..****X | 9522.000 | | 7206.500 | ..........................................*X | 7206.500 | | 7206.500 | ..........................................*...******...............*X | 10193.000 | | 9629.900 | .........................................................******.....**...*****X | 10923.200 | | 6895.500 | ........................................*X | 6895.500 | | 5647.100 | .................................******...***.*X | 6301.800 | | 6301.800 | .....................................**X | 6301.800 | | 4239.200 | ........................**X | 4239.200 | | 3600.500 | ....................*****X | 3600.500 | | 1470.400 | .......**.******....................................................**.................******......................*X | 16448.200 | | 15909.800 | ...............................................................................................*****X | 15909.800 | | 10413.100 | ..............................................................**X | 10413.100 | | 9921.000 | ...........................................................****X | 9921.000 | | 7984.400 | ...............................................**X | 7984.400 | | 7008.800 | .........................................****X | 7008.800 | | 7008.800 | .........................................X | 7008.800 | | 6015.700 | ...................................*X | 6015.700 | | 5659.200 | .................................*X | 5659.200 | | 5659.200 | .................................*X | 5659.200 | | 5659.200 | .................................**..**....******........................**.....*****X | 11402.100 | | 9562.600 | .........................................................**X | 9562.600 | | 9127.000 | ......................................................*****X | 9127.000 | | 6881.300 | ........................................*X | 6881.300 | | 6177.900 | ....................................*****X | 6177.900 | | 4958.000 | .............................*X | 4958.000 | | 4725.000 | ...........................******...*X | 5107.600 | | 4789.500 | ............................******.***....*****......**X | 6705.000 | | 6377.500 | .....................................*****X | 6377.500 | | 4237.800 | ........................*X | 4237.800 | | 4077.500 | .......................******.......**...******..........**X | 7270.700 | | 7264.900 | ...........................................*****..***..*****X | 8006.600 | | 7135.400 | ..........................................X | 7135.400 | | 7135.400 | ..........................................X | 7135.400 | | 6562.900 | ......................................*X | 6562.900 | | 6562.900 | ......................................*X | 6562.900 | | 6562.900 | ......................................*X | 6562.900 | | 4665.100 | ...........................*X | 4665.100 | | 4631.000 | ...........................******......*X | 5667.600 | | 5580.400 | ................................******..***.******...........**...X | 8358.900 | | 8358.900 | .................................................**X | 8358.900 | | 8358.900 | .................................................*X | 8358.900 | | 7439.800 | ............................................*X | 7439.800 | | 7197.500 | ..........................................******....***.....****X | 8576.600 | | 8538.600 | ..................................................*X | 8538.600 | | 8538.600 | ..................................................X | 8538.600 | | 7590.900 | .............................................*****X | 7590.900 | | 5063.900 | .............................*X | 5063.900 | | 4590.000 | ..........................******................**..........******............**X | 10831.100 | | 10816.600 | ................................................................****X | 10816.600 | | 7470.500 | ............................................**X | 7470.500 | | 6377.700 | .....................................*****X | 6377.700 | | 5837.100 | ..................................**..******.................*X | 8906.900 | | 8846.100 | ....................................................X | 8846.100 | | 8846.100 | ....................................................***X | 8846.100 | | 8846.100 | ....................................................X | 8846.100 | | 6397.600 | .....................................*X | 6397.600 | | 6339.200 | .....................................******.............***.........*****..........................***..*****X | 14548.700 | | 14087.700 | ....................................................................................**.*****X | 14249.500 | | 13021.800 | ..............................................................................*X | 13021.800 | | 12167.000 | ........................................................................*****X | 12167.000 | | 11399.400 | ....................................................................**X | 11399.400 | | 10603.000 | ...............................................................****X | 10603.000 | | 8305.100 | .................................................**X | 8305.100 | | 8000.900 | ...............................................*****X | 8000.900 | | 5996.200 | ...................................********..................................**.......******.*X | 12872.900 | | 12422.200 | ..........................................................................*****X | 12422.200 | | 6083.700 | ...................................**X | 6083.700 | | 4878.400 | ............................*****..............................X | 9727.600 | | 9727.600 | ..........................................................**.............*****X | 11872.900 | | 7820.300 | ..............................................*X | 7820.300 | | 5759.900 | ..................................******................**..*****X | 8851.200 | | 5912.700 | ..................................***..******............X | 8090.700 | | 8090.700 | ................................................X | 8090.700 | | 7413.300 | ............................................*****X | 7413.300 | | 2430.300 | .............*X | 2430.300 | | 1518.500 | ........*********.*****X | 1701.100 | | 1253.200 | ......*X | 1253.200 | | 1209.000 | ......******...........**...******......................**.......******..**X | 8707.800 | | 8228.700 | .................................................****X | 8228.700 | | 5443.400 | ................................**X | 5443.400 | | 4678.300 | ...........................******.........**......******.........**..*****X | 9024.700 | | 3890.100 | ......................*X | 3890.100 | | 2198.400 | ............******...........................**............*****X | 8581.200 | | 7045.400 | .........................................**X | 7045.400 | | 6196.400 | ....................................*****.......**X | 7279.300 | | 7240.100 | ...........................................******.*X | 7549.100 | | 7376.300 | ...........................................*****X | 7376.300 | | 5234.600 | ..............................**X | 5234.600 | | 4824.200 | ............................****X | 4824.200 | | 4048.500 | .......................*X | 4048.500 | | 3888.600 | ......................******........**..******.....***.******..........**.......*****X | 9213.700 | | 8436.100 | ..................................................**...******.....................**.*****X | 12520.800 | | 12241.000 | .........................................................................**X | 12241.000 | | 12164.000 | ........................................................................****X | 12164.000 | | 10019.400 | ...........................................................**X | 10019.400 | | 9278.500 | .......................................................*****X | 9278.500 | | 9043.500 | .....................................................*X | 9043.500 | | 8192.400 | ................................................*****X | 8192.400 | | 7128.600 | ..........................................**....******..*X | 8068.900 | | 7693.100 | .............................................******....................***...*****X | 11403.300 | | 10443.300 | ..............................................................*X | 10443.300 | | 10074.400 | ............................................................*****X | 10074.400 | | 6892.400 | ........................................*X | 6892.400 | | 6792.100 | ........................................******...........................***...****X | 11703.300 | | 7845.500 | ..............................................**X | 7845.500 | | 7443.800 | ............................................*****X | 7443.800 | | 6959.700 | .........................................*X | 6959.700 | | 6318.600 | .....................................*****X | 6318.600 | | 4746.900 | ...........................*X | 4746.900 | | 4199.500 | ........................*****X | 4199.500 | | 1819.900 | ..........*X | 1819.900 | | 1295.200 | ......******..............***.......*****..........................***...****X | 9436.200 | | 8916.000 | .....................................................***..******...**...*****X | 10287.300 | | 10125.000 | ............................................................*X | 10125.000 | | 9370.300 | .......................................................*****X | 9370.300 | | 9313.900 | .......................................................***...*****........**X | 11138.000 | | 10703.300 | ................................................................*****X | 10703.300 | | 9467.400 | ........................................................*X | 9467.400 | | 8940.100 | .....................................................*****X | 8940.100 | | 7119.500 | ..........................................*X | 7119.500 | | 7041.200 | .........................................******...............**.......*****X | 10597.600 | | 8692.600 | ...................................................*X | 8692.600 | | 7484.100 | ............................................******............***.....*****X | 10295.600 | | 7884.100 | ..............................................*X | 7884.100 | | 7251.400 | ...........................................*****X | 7251.400 | | 7228.800 | ..........................................*X | 7228.800 | | 7078.600 | ..........................................******.......**.*****X | 8552.700 | | 8232.400 | .................................................**....******...................**X | 12161.700 | | 11817.800 | ......................................................................*****X | 11817.800 | | 5718.900 | .................................*X | 5718.900 | | 4525.300 | ..........................*****X | 4525.300 | | 3494.100 | ....................*X | 3494.100 | | 3394.900 | ...................*****X | 3394.900 | | 3168.500 | ..................*X | 3168.500 | | 2884.300 | ................*****X | 2884.300 | | 2791.900 | ...............**.*****X | 2824.100 | | 2697.800 | ...............*********....**.******.....**...******....*X | 5538.000 | | 5283.900 | ...............................******.***..*****X | 5905.800 | | 5862.600 | ..................................**.*****X | 5941.600 | | 4809.000 | ............................**.....*****X | 5663.600 | | 3729.900 | .....................*X | 3729.900 | | 2201.400 | ............*****X | 2201.400 | | 1141.300 | .....**X | 1141.300 | | 382.600 | .********.******..**..*****X | 1271.500 | | 1171.100 | ......*X | 1171.100 | | 792.200 | ...*****X | 792.200 | | 427.400 | .***.******...**.*****X | 1225.700 | | 873.800 | ....*X | 873.800 | | 657.900 | ..******.***...******.........................********...**..******................**.....******......................**....*****X | 13946.300 | | 10149.400 | ............................................................**X | 10149.400 | | 8845.400 | ....................................................****X | 8845.400 | | 7198.100 | ..........................................*...*X | 7711.400 | | 6828.900 | ........................................******.............**......*****X | 9966.100 | | 5861.200 | ..................................*X | 5861.200 | | 4791.000 | ............................******...............***......*X | 8227.000 | +-----------+------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 05:50:40,762 | test | INFO | pool-5-thread-1 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------+----------+ | 8227.000 | ...................................................................................................X | 8227.000 | +----------+------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 05:50:45,111 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 96.3 2022-02-02 05:50:55,148 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 96.47 2022-02-02 05:51:05,223 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 96.74 2022-02-02 05:51:15,262 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 97.83 2022-02-02 05:51:25,301 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: running, progress: 99.4 2022-02-02 05:51:40,348 | test | INFO | pool-5-thread-5 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 05:51:40,388 | test | INFO | pool-5-thread-5 | [task:check:468] Rebalance completed with progress: 100% in 1632.31299996 sec 2022-02-02 05:51:40,516 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.105.206']} 2022-02-02 05:51:45,960 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 05:51:58,280 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 05:51:58,282 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 05:52:07,108 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 05:52:07,108 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 05:52:16,049 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 05:52:16,049 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 05:52:30,950 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 92100 | 7.81 GiB | 289.41 MiB | 361.07 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 3.91 GiB | 490.51 MiB | 506.66 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 15010750 | 2.00 GiB | 1.35 GiB | 28.79 GiB | 28.7889146112 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 05:52:31,441 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 05:52:31,443 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 05:52:37,829 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 05:52:37,829 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 05:52:44,098 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 26% 2022-02-02 05:52:44,099 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 6% 2022-02-02 05:53:37,641 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 05:53:38,101 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 05:53:38,102 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 05:54:08,233 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 05:54:08,653 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 05:54:08,654 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 05:54:38,756 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 05:54:39,183 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 05:54:39,183 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 05:57:40,611 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9051ed1400ab3d42682ef9fe3332b73e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c011d97035e9c7800af8bd4018b67e0c', u'status': u'notRunning'} - rebalance failed 2022-02-02 05:57:40,641 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 05:57:40,641 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643810259376L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:57:39.376Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 670a82c1e932d673106e6ad080b81479'} 2022-02-02 05:57:40,642 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643810259375L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:57:39.375Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 05:57:40,642 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643810080058L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:54:40.058Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.105.36\' for deletion'} 2022-02-02 05:57:40,642 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643810079372L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:54:39.372Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 670a82c1e932d673106e6ad080b81479"} 2022-02-02 05:57:40,642 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643810079365L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T05:54:39.365Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\']'} 2022-02-02 05:57:40,644 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643810024623L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:53:44.623Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.105.36\' for deletion'} 2022-02-02 05:57:40,644 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643810023895L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:53:43.895Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 05:57:40,644 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643810005393L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:53:25.393Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 05:57:40,644 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643810002995L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:53:22.995Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 05:57:40,644 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643809987745L, u'shortText': u'message', u'serverTime': u'2022-02-02T05:53:07.745Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes 2022-02-02 05:57:40,645 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9051ed1400ab3d42682ef9fe3332b73e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c011d97035e9c7800af8bd4018b67e0c', u'status': u'notRunning'} - rebalance failed return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9051ed1400ab3d42682ef9fe3332b73e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c011d97035e9c7800af8bd4018b67e0c', u'status': u'notRunning'} - rebalance failed During the test, Remote Connections: 241, Disconnections: 241 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_2 ok ---------------------------------------------------------------------- Ran 1 test in 6809.832s OK Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_full_recovery_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_3', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_full_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 3, 'randomize_value': 'True'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 05:57:47,578 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 05:57:48,157 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 05:58:57,530 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 68.6019999981 seconds to complete 2022-02-02 05:58:57,539 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 05:59:02,611 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x1addf2 type=client open_count=1 channel=[id: 0x22fd99a3, 0.0.0.0/0.0.0.0:38645] timeout=300.0>) 2022-02-02 05:59:05,641 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x1addf3 type=client open_count=1 channel=[id: 0x342d9062, 0.0.0.0/0.0.0.0:49425] timeout=300.0>) 2022-02-02 05:59:07,653 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x1addf4 type=client open_count=1 channel=[id: 0x416ada98, 0.0.0.0/0.0.0.0:43899] timeout=300.0>) 2022-02-02 05:59:09,667 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x1addf5 type=client open_count=1 channel=[id: 0xc6286594, 0.0.0.0/0.0.0.0:55471] timeout=300.0>) 2022-02-02 05:59:11,677 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x1addf6 type=client open_count=1 channel=[id: 0x88113213, 0.0.0.0/0.0.0.0:39773] timeout=300.0>) 2022-02-02 05:59:17,395 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x1addf7 type=client open_count=1 channel=[id: 0x99122992, 0.0.0.0/0.0.0.0:58757] timeout=300.0>) 2022-02-02 05:59:19,407 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x1addf8 type=client open_count=1 channel=[id: 0xbde39627, 0.0.0.0/0.0.0.0:46815] timeout=300.0>) 2022-02-02 05:59:21,421 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x1addf9 type=client open_count=1 channel=[id: 0x8fc53cef, 0.0.0.0/0.0.0.0:60426] timeout=300.0>) 2022-02-02 05:59:23,433 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x1addfa type=client open_count=1 channel=[id: 0xcbd90cd2, 0.0.0.0/0.0.0.0:54561] timeout=300.0>) 2022-02-02 05:59:25,444 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x1addfb type=client open_count=1 channel=[id: 0x59d07315, 0.0.0.0/0.0.0.0:53926] timeout=300.0>) 2022-02-02 05:59:31,045 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x1addfc type=client open_count=1 channel=[id: 0x044b20f7, 0.0.0.0/0.0.0.0:55955] timeout=300.0>) 2022-02-02 05:59:33,056 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x1addfd type=client open_count=1 channel=[id: 0xa231cbb1, 0.0.0.0/0.0.0.0:41053] timeout=300.0>) 2022-02-02 05:59:35,071 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x1addfe type=client open_count=1 channel=[id: 0xa299c419, 0.0.0.0/0.0.0.0:47733] timeout=300.0>) 2022-02-02 05:59:37,084 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x1addff type=client open_count=1 channel=[id: 0xe38b99b8, 0.0.0.0/0.0.0.0:50261] timeout=300.0>) 2022-02-02 05:59:39,104 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1ade00 type=client open_count=1 channel=[id: 0x825104c4, 0.0.0.0/0.0.0.0:42295] timeout=300.0>) 2022-02-02 05:59:45,335 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1ade01 type=client open_count=1 channel=[id: 0x0a0bc295, 0.0.0.0/0.0.0.0:43585] timeout=300.0>) 2022-02-02 05:59:47,351 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1ade02 type=client open_count=1 channel=[id: 0x5a99952a, 0.0.0.0/0.0.0.0:40295] timeout=300.0>) 2022-02-02 05:59:49,361 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x1ade03 type=client open_count=1 channel=[id: 0x475fae06, 0.0.0.0/0.0.0.0:46023] timeout=300.0>) 2022-02-02 05:59:51,377 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1ade04 type=client open_count=1 channel=[id: 0x16aa4e94, 0.0.0.0/0.0.0.0:36159] timeout=300.0>) 2022-02-02 06:00:00,207 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1ade05 type=client open_count=1 channel=[id: 0x14f23ace, 0.0.0.0/0.0.0.0:56533] timeout=300.0>) 2022-02-02 06:00:02,237 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1ade06 type=client open_count=1 channel=[id: 0x950dbbe8, 0.0.0.0/0.0.0.0:53661] timeout=300.0>) 2022-02-02 06:00:04,252 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1ade07 type=client open_count=1 channel=[id: 0xfd918d46, 0.0.0.0/0.0.0.0:59236] timeout=300.0>) 2022-02-02 06:00:06,265 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x1ade08 type=client open_count=1 channel=[id: 0x805ae90b, 0.0.0.0/0.0.0.0:52134] timeout=300.0>) 2022-02-02 06:00:08,276 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x1ade09 type=client open_count=1 channel=[id: 0xbb2c80fc, 0.0.0.0/0.0.0.0:55831] timeout=300.0>) 2022-02-02 06:00:13,980 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x1ade0a type=client open_count=1 channel=[id: 0x378051fc, 0.0.0.0/0.0.0.0:39303] timeout=300.0>) 2022-02-02 06:00:15,993 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x1ade0b type=client open_count=1 channel=[id: 0xd5e947e2, 0.0.0.0/0.0.0.0:44370] timeout=300.0>) 2022-02-02 06:00:18,005 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x1ade0c type=client open_count=1 channel=[id: 0x5b36c913, 0.0.0.0/0.0.0.0:51702] timeout=300.0>) 2022-02-02 06:00:20,016 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x1ade0d type=client open_count=1 channel=[id: 0xb5772dd1, 0.0.0.0/0.0.0.0:46791] timeout=300.0>) 2022-02-02 06:00:22,040 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x1ade0e type=client open_count=1 channel=[id: 0x93cd4a9f, 0.0.0.0/0.0.0.0:52334] timeout=300.0>) 2022-02-02 06:00:27,388 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x1ade0f type=client open_count=1 channel=[id: 0x09ce5101, 0.0.0.0/0.0.0.0:59533] timeout=300.0>) 2022-02-02 06:00:29,401 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x1ade10 type=client open_count=1 channel=[id: 0xa37aaddb, 0.0.0.0/0.0.0.0:51589] timeout=300.0>) 2022-02-02 06:00:31,415 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x1ade11 type=client open_count=1 channel=[id: 0xa9637211, 0.0.0.0/0.0.0.0:37421] timeout=300.0>) 2022-02-02 06:00:33,430 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 06:00:50,697 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 06:01:03,641 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 06:01:03,642 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 06:01:03,664 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 06:01:16,723 | test | INFO | pool-6-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.66541447533 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 06:01:16,749 | test | INFO | pool-6-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 06:01:31,790 | test | INFO | pool-6-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 06:01:31,805 | test | INFO | pool-6-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0820000172 sec 2022-02-02 06:01:31,938 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.351229302559 | 11.45 GiB | 10.68 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.250563768479 | 11.45 GiB | 10.69 GiB | 24.25 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.10358665663 | 11.45 GiB | 10.64 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.225535647162 | 11.45 GiB | 10.76 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 0.914901616744 | 11.45 GiB | 10.62 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 06:01:31,940 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 06:01:31,940 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 06:01:32,217 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 06:01:32,882 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 06:01:33,187 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 10700 | 4.88 GiB | 113.05 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ 2022-02-02 06:01:33,187 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 06:01:43,813 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 06:01:43,822 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 06:51:31,095 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...................******.................................................................................*X | 750.000 | | 604.500 | ...............................................................................*****X | 604.500 | | 0.000 | ************************************************************************************************************************************************************************************************************************..........**...********.******......**X | 150.000 | | 140.300 | .................****X | 140.300 | | 120.500 | ...............X | 120.500 | | 59.700 | ......*X | 59.700 | | 50.000 | .....*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *************************************************************************************************************************************************************************************************************************************................**....*****X | 150.000 | | 69.400 | ........*X | 69.400 | | 50.000 | .....*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *************************************************************************************************************************************************************************************.....**.**************.....................***.....*****X | 241.900 | | 56.600 | ......*X | 56.600 | | 8.100 | *****X | 8.100 | | 0.000 | *********.....**.**************......***...*******X | 119.200 | | 80.800 | .........*****X | 80.800 | | 0.200 | *X | 0.200 | | 0.000 | ******************************************************************************************************************************************************************************************************************************.....**.******......**..********X | 109.700 | | 90.300 | ...........******..........**....*****X | 200.000 | | 119.400 | ..............*X | 119.400 | | 100.000 | ............*************X | 100.000 | | 19.400 | .**X | 19.400 | | 0.000 | ***********************************************************************************************************************************************************************************************************......................***....*****X | 195.900 | | 75.000 | .........*X | 75.000 | | 50.000 | .....*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ********************************************************************************************************************************************************************************************************************..........**...*****X | 100.000 | | 19.400 | .*X | 19.400 | | 0.000 | *****************.******................**....*******X | 159.700 | | 130.700 | ................******.....**....*****X | 200.000 | | 38.800 | ....********................................**X | 280.600 | | 261.200 | .................................*****X | 261.200 | | 19.400 | .*X | 19.400 | | 0.000 | ******..........**...*****X | 100.000 | | 19.400 | .**X | 19.400 | | 0.000 | *********************************************************************************************************************************************************************************************.*****.....**X | 50.000 | | 40.300 | ....********.******......**.*****X | 100.000 | | 19.400 | .*X | 19.400 | | 0.000 | **************.....***.****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | **********************************************************************************************************************************************************************************************************************.******.....*X | 50.000 | | 40.300 | ....*********.********.******......*X | 100.000 | | 98.100 | ............********.********X | 110.600 | | 81.600 | .........****X | 81.600 | | 1.000 | **X | 1.000 | | 0.000 | ******.....**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *******************************************************************************************************************************************************************.....***.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | **************................***...****X | 146.300 | | 25.400 | ..**X | 25.400 | | 9.700 | ******.....*X | 50.000 | | 48.100 | .....******.....**.*****X | 92.200 | | 11.600 | **X | 11.600 | | 0.000 | ***************************************************************************************************************************************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 06:51:31,980 | test | INFO | pool-6-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 426.800 | ..................******.................................................................................**X | 2200.000 | | 1773.200 | ...............................................................................*****X | 1773.200 | | 0.000 | ************************************************************************************************************************************************************************************************************************...**..********.******...........**X | 380.600 | | 341.800 | ..............****X | 341.800 | | 182.600 | .......X | 182.600 | | 19.400 | *X | 19.400 | | 0.000 | ************************************************************************************************************************************************************************************************************************************.**.******........********X | 240.300 | | 79.100 | ..*X | 79.100 | | 50.000 | .*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *************************************************************************************************************************************************************************************************************.***.******..***************X | 100.000 | | 59.700 | .**X | 59.700 | | 50.000 | .*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *******************************************************************************************************************************************************************************************************************************************************.......**..*************X | 200.000 | | 38.800 | **X | 38.800 | | 0.000 | **************..*X | 50.000 | | 40.300 | *****X | 40.300 | | 0.000 | **********************************************************************************************************************************************************************************************....*******X | 90.300 | | 9.700 | *X | 9.700 | | 0.000 | ******.***.****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ***********************************************************************************************************************************************************************************************************.***.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ***************************************.....**.*****X | 150.000 | | 69.400 | ..*X | 69.400 | | 40.300 | ******..**.*****X | 100.000 | | 19.400 | **X | 19.400 | | 9.700 | *****..**X | 50.000 | | 40.300 | *****X | 40.300 | | 0.000 | *****************************************************************************************************************************************************************************************************************************...***.*****X | 100.000 | | 59.700 | .*X | 59.700 | | 50.000 | .*************X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ********************************************************.**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ***************************************************************************************************************************************************************************..*X | 50.000 | | 40.300 | *****X | 40.300 | | 0.000 | *************************....*X | 100.000 | | 80.600 | ..*****X | 80.600 | | 40.300 | ***.****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ******.**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ********************************************************************************************************************************************************************************************.***.****X | 50.000 | | 19.600 | X | 19.600 | | 9.700 | *X | 9.700 | | 0.000 | *****************************************************************************************************************************************************************X | 0.000 | +----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 06:51:32,441 | test | INFO | pool-6-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******..............................................................................**......................*****X | 35402.600 | | 21910.600 | ............................................................**X | 21910.600 | | 16506.800 | .............................................*****X | 16506.800 | | 7096.100 | ...................*X | 7096.100 | | 6566.400 | .................******.........**X | 9671.000 | | 9652.800 | ..........................*****X | 9652.800 | | 8075.800 | .....................********.*X | 8177.300 | | 7937.600 | .....................******..***.*****...**X | 10110.300 | | 10010.800 | ...........................********...******...............**..******....*X | 18423.100 | | 17944.200 | .................................................*****X | 17944.200 | | 10840.500 | .............................*X | 10840.500 | | 10074.900 | ...........................******....*********..**.*****X | 12413.700 | | 11625.300 | ...............................*X | 11625.300 | | 11303.500 | ..............................*****X | 11303.500 | | 9519.400 | .........................***...*****.................**X | 16521.500 | | 15815.400 | ...........................................*****X | 15815.400 | | 9146.100 | ........................*X | 9146.100 | | 8862.900 | ........................******....*******X | 10596.600 | | 8893.200 | ........................**X | 8893.200 | | 8691.600 | .......................*****..***.******.*X | 9935.300 | | 9441.700 | .........................******.**.*****X | 10122.800 | | 9554.800 | .........................**X | 9554.800 | | 9455.300 | .........................******.....**.*****X | 11616.000 | | 10909.800 | .............................*X | 10909.800 | | 10614.200 | ............................*****X | 10614.200 | | 8749.400 | .......................*X | 8749.400 | | 8099.000 | .....................******.***..******...**.******......*******X | 12631.000 | | 8763.400 | .......................*X | 8763.400 | | 8132.200 | .....................******.....********X | 9877.800 | | 7491.300 | ....................*X | 7491.300 | | 7283.500 | ...................******.********.................**..*****X | 14489.200 | | 9014.700 | ........................**X | 9014.700 | | 8258.100 | ......................******....*******X | 9637.800 | | 9095.200 | ........................*X | 9095.200 | | 8947.900 | ........................******.....**..******.....***.****X | 13529.700 | | 12375.700 | .................................**X | 12375.700 | | 11673.600 | ...............................******..**.******...*X | 13551.000 | | 13523.100 | .....................................*****X | 13523.100 | | 12234.000 | .................................*X | 12234.000 | | 11960.900 | ................................******.....***...****X | 14643.200 | | 14430.100 | .......................................**X | 14430.100 | | 13775.800 | .....................................*****X | 13775.800 | | 12322.700 | .................................********.............**..*****X | 17589.000 | | 14676.200 | ........................................*X | 14676.200 | | 14163.800 | .......................................******.**X | 14670.300 | | 14489.800 | .......................................****X | 14489.800 | | 11254.000 | ..............................**X | 11254.000 | | 10595.600 | ............................*****X | 10595.600 | | 9883.300 | ..........................*X | 9883.300 | | 9731.600 | ..........................******.**.*****X | 10303.600 | | 7488.400 | ....................*X | 7488.400 | | 6503.700 | .................*****X | 6503.700 | | 6442.300 | .................********...***.*****X | 7858.900 | | 7450.000 | ....................*X | 7450.000 | | 7408.600 | ...................******...**.******............**...******.........**...******......**X | 20363.400 | | 19161.000 | .....................................................****X | 19161.000 | | 14642.300 | ........................................********X | 14660.100 | | 13576.900 | .....................................*X | 13576.900 | | 12785.900 | ...................................*****X | 12785.900 | | 7887.700 | .....................*X | 7887.700 | | 6687.700 | .................******...........**....******.....***.****X | 13832.800 | | 13575.200 | .....................................**X | 13575.200 | | 13407.400 | ....................................*****X | 13407.400 | | 12422.400 | ..................................*X | 12422.400 | | 12120.000 | .................................********.*****X | 12744.600 | | 12210.000 | .................................*X | 12210.000 | | 11722.300 | ................................*************X | 12031.600 | | 9836.400 | ..........................**X | 9836.400 | | 9108.200 | ........................******..........**...*****X | 13659.900 | | 12458.400 | ..................................*X | 12458.400 | | 12052.700 | .................................**************.......**.*****X | 15018.400 | | 12558.800 | ..................................**X | 12558.800 | | 12107.100 | .................................****X | 12107.100 | | 11323.800 | ..............................***.*****.....*..*******X | 13886.500 | | 13396.700 | ....................................**.*******X | 13800.700 | | 13692.400 | .....................................*****X | 13692.400 | | 13331.300 | ....................................**.*****X | 13638.000 | | 12184.300 | .................................**X | 12184.300 | | 10960.000 | .............................****X | 10960.000 | | 10280.000 | ............................X | 10280.000 | | 8206.600 | ......................*X | 8206.600 | | 8197.100 | ......................*****X | 8197.100 | | 6863.800 | ..................*X | 6863.800 | | 6352.800 | ................******..**.******.**X | 7719.300 | | 7435.200 | ....................*****X | 7435.200 | | 7319.300 | ...................**..******.*X | 8250.700 | | 8044.900 | .....................*****X | 8044.900 | | 7076.700 | ..................**X | 7076.700 | | 6967.700 | ..................****X | 6967.700 | | 6836.200 | ..................**X | 6836.200 | | 6487.600 | .................*****X | 6487.600 | | 5801.700 | ...............**.******...**.******......**...******.........**..******...*******X | 15824.900 | | 14489.400 | .......................................**X | 14489.400 | | 13634.500 | .....................................*****X | 13634.500 | | 10168.900 | ...........................**.******.......**.******.********......***.****X | 16030.800 | | 13452.300 | ....................................**X | 13452.300 | | 13236.300 | ....................................******....*X | 14683.200 | | 14681.100 | ........................................*****X | 14681.100 | | 11649.700 | ...............................*X | 11649.700 | | 11200.000 | ..............................********X | 11264.800 | | 10696.100 | .............................****X | 10696.100 | | 7845.200 | .....................**X | 7845.200 | | 7805.400 | .....................*****...**X | 9169.200 | | 9141.600 | ........................*****X | 9141.600 | | 7459.000 | ....................*X | 7459.000 | | 7293.600 | ...................******..**.*****X | 8272.600 | | 7842.000 | .....................*X | 7842.000 | | 7425.900 | ...................*****X | 7425.900 | | 5767.900 | ...............**X | 5767.900 | | 5508.600 | ..............*****X | 5508.600 | | 3290.000 | ........**.******...*X | 4837.100 | | 4786.200 | ............******......***.*****.....***..*****..........***.*****X | 13701.600 | | 12022.900 | ................................*X | 12022.900 | | 11401.200 | ...............................*****X | 11401.200 | | 10283.900 | ............................**.******....**.*****X | 12426.200 | | 11535.200 | ...............................**X | 11535.200 | | 10403.100 | ............................*****X | 10403.100 | | 8680.700 | .......................**..******........**.*****X | 12649.200 | | 9125.100 | ........................*X | 9125.100 | | 8112.900 | .....................******...***..*****......**X | 11823.100 | | 11419.600 | ...............................*****X | 11419.600 | | 6943.500 | ..................*X | 6943.500 | | 6269.500 | ................******.**.******.............***...*****.**X | 13030.100 | | 12974.000 | ...................................*****X | 12974.000 | | 11468.400 | ...............................*X | 11468.400 | | 11212.000 | ..............................******......**.*****X | 13669.700 | | 13155.800 | ....................................*X | 13155.800 | | 12563.000 | ..................................*********..******.*X | 13772.100 | | 13176.700 | ....................................*****X | 13176.700 | | 12737.400 | ..................................**..*****X | 13145.800 | | 13011.100 | ...................................*X | 13011.100 | | 12617.000 | ..................................*****X | 12617.000 | | 12450.500 | ..................................********X | 12482.600 | | 10884.800 | .............................*X | 10884.800 | | 10569.800 | ............................*****X | 10569.800 | | 10409.900 | ............................********....*X | 11688.800 | | 11618.100 | ...............................******...***..*******X | 13343.500 | | 13162.800 | ....................................*****X | 13162.800 | | 12159.000 | .................................*X | 12159.000 | | 11941.500 | ................................*****X | 11941.500 | | 11269.700 | ..............................********..*X | 11807.600 | | 11564.900 | ...............................*****X | 11564.900 | | 11289.500 | ..............................***.*****.**X | 12023.100 | | 11710.800 | ................................*****X | 11710.800 | | 10515.200 | ............................**.******.........**.*****X | 14310.600 | | 10089.500 | ...........................**X | 10089.500 | | 9047.100 | ........................*****........***...******...**.*****X | 14166.400 | | 13354.900 | ....................................*X | 13354.900 | | 12962.700 | ...................................******...*******X | 13979.200 | | 13677.000 | .....................................*X | 13677.000 | | 13524.300 | .....................................*****X | 13524.300 | | 11738.100 | ................................**X | 11738.100 | | 11476.800 | ...............................*****.***.******.........**..*****X | 16028.000 | | 14573.800 | ........................................*X | 14573.800 | | 14185.100 | .......................................*****X | 14185.100 | | 13679.900 | .....................................*X | 13679.900 | | 13386.900 | ....................................*****X | 13386.900 | | 11213.400 | ..............................**X | 11213.400 | | 11101.300 | ..............................*****...........***..*****X | 15849.600 | | 11868.700 | ................................*X | 11868.700 | | 10872.400 | .............................*****X | 10872.400 | | 10432.000 | ............................*X | 10432.000 | | 10422.400 | ............................*****X | 10422.400 | | 9454.400 | .........................**X | 9454.400 | | 9200.200 | ........................******...**.*****X | 10309.700 | | 9232.500 | .........................*X | 9232.500 | | 9118.500 | ........................*****X | 9118.500 | | 8742.000 | .......................**.******.*X | 9233.700 | | 8969.200 | ........................*****X | 8969.200 | | 8803.200 | .......................**X | 8803.200 | | 8697.300 | .......................****X | 8697.300 | | 8286.100 | ......................**X | 8286.100 | | 8160.400 | ......................**************..**.******.*******X | 9807.200 | | 8424.000 | ......................*X | 8424.000 | | 7910.500 | .....................******...**.*****X | 9367.000 | | 8372.100 | ......................********....*******X | 9666.000 | | 9206.700 | .........................********X | 9359.800 | | 9131.300 | ........................*X | 9131.300 | | 8726.000 | .......................*****X | 8726.000 | | 7918.500 | .....................********.....**.*****X | 9990.300 | | 7473.200 | ....................**X | 7473.200 | | 6948.300 | ..................*****..*********..**.*****X | 8559.400 | | 7896.800 | .....................*X | 7896.800 | | 7698.100 | ....................******.*X | 7814.200 | | 7642.700 | ....................******.**.*****X | 8296.300 | | 6451.500 | .................**X | 6451.500 | | 5957.300 | ...............*****.*.**.******......**.*****X | 9396.600 | | 9167.100 | ........................*X | 9167.100 | | 8973.000 | ........................******..**.******..**X | 10651.600 | | 10379.800 | ............................*****X | 10379.800 | | 9926.200 | ...........................*******X | 10070.800 | | 8331.000 | ......................*X | 8331.000 | | 7720.800 | ....................******.........**...*****X | 11689.700 | | 10322.300 | ............................**X | 10322.300 | | 10286.300 | ............................*****......***..******.*X | 13663.000 | | 13468.000 | .....................................*****X | 13468.000 | | 11583.100 | ...............................*X | 11583.100 | | 11300.300 | ..............................******.*X | 11330.400 | | 11136.600 | ..............................*****X | 11136.600 | | 8797.200 | .......................**X | 8797.200 | | 8423.600 | ......................*****.....***.*****X | 10418.700 | | 10096.600 | ...........................*X | 10096.600 | | 9669.000 | ..........................*****X | 9669.000 | | 7529.800 | ....................*X | 7529.800 | | 7150.900 | ...................******....***..*****X | 9546.100 | | 9001.700 | ........................*X | 9001.700 | | 8305.700 | ......................*****X | 8305.700 | | 8109.300 | .....................**..******.........**.*****X | 12329.900 | | 10021.300 | ...........................*X | 10021.300 | | 9329.900 | .........................*****X | 9329.900 | | 6995.800 | ..................*X | 6995.800 | | 6513.200 | .................******..**..******......**X | 10116.600 | | 9885.700 | ..........................*****.***.****X | 10384.100 | | 8734.700 | .......................**X | 8734.700 | | 8068.500 | .....................******....**..******..*X | 10912.200 | | 10676.400 | .............................*****X | 10676.400 | | 9745.000 | ..........................*******X | 9779.500 | | 8114.200 | .....................*X | 8114.200 | | 7627.500 | ....................******..**.******...***.*****..**X | 10644.900 | | 10644.400 | .............................******...**.******.*X | 12559.000 | | 12390.300 | .................................******..**.*****X | 13183.900 | | 12001.700 | ................................*X | 12001.700 | | 11576.400 | ...............................******......*******X | 13764.200 | | 11655.800 | ...............................**X | 11655.800 | | 11406.500 | ...............................****X | 11406.500 | | 10181.600 | ...........................**X | 10181.600 | | 9665.400 | ..........................*****X | 9665.400 | | 9062.500 | ........................**.******.*X | 9568.400 | | 8920.400 | ........................*****X | 8920.400 | | 7100.800 | ...................********..**X | 8032.400 | | 8014.300 | .....................*****X | 8014.300 | | 7099.400 | ...................*******X | 7100.900 | | 7092.400 | ...................*X | 7092.400 | | 7083.500 | ...................*****X | 7083.500 | | 2204.800 | .....*X | 2204.800 | | 1770.300 | ....******..........***.*****..............***...*****X | 11789.700 | | 11346.100 | ...............................*X | 11346.100 | | 11161.400 | ..............................******...........**.*****X | 15316.000 | | 12096.600 | .................................*X | 12096.600 | | 11749.400 | ................................*********.*****......***.******..**.*****X | 15609.200 | | 14854.900 | ........................................**.*****X | 14903.400 | | 12522.400 | ..................................**X | 12522.400 | | 11900.900 | ................................*****..***.*****X | 13006.300 | | 10784.600 | .............................*X | 10784.600 | | 10202.500 | ...........................******..........**.*****X | 13895.000 | | 12511.900 | ..................................*******X | 12566.000 | | 10601.800 | ............................**X | 10601.800 | | 10404.200 | ............................******...........**.******..*******X | 15564.300 | | 12315.300 | .................................**X | 12315.300 | | 11584.800 | ...............................*****.*********....**.*****X | 13471.700 | | 10669.200 | .............................*X | 10669.200 | | 10356.000 | ............................******........***..*****.***.*****X | 14805.400 | | 12939.000 | ...................................*X | 12939.000 | | 12160.400 | .................................*****X | 12160.400 | | 11557.700 | ...............................*X | 11557.700 | | 11254.800 | ..............................*****X | 11254.800 | | 11172.400 | ..............................***..*****....**X | 13128.500 | | 13108.300 | ....................................*******X | 13328.100 | | 13187.600 | ....................................*****X | 13187.600 | | 11235.600 | ..............................*X | 11235.600 | | 11044.100 | ..............................******..*******X | 11838.700 | | 9069.500 | ........................**X | 9069.500 | | 8531.400 | .......................*****...*..........**..*****X | 14123.600 | | 13803.900 | .....................................*X | 13803.900 | | 13634.100 | .....................................*****X | 13634.100 | | 11820.300 | ................................*X | 11820.300 | | 11085.600 | ..............................*****X | 11085.600 | | 10541.200 | ............................***..******......*******X | 13239.800 | | 11564.600 | ...............................*X | 11564.600 | | 11230.700 | ..............................*****X | 11230.700 | | 10406.300 | ............................********....**X | 11834.500 | | 11580.200 | ...............................*****X | 11580.200 | | 9164.100 | ........................*X | 9164.100 | | 8759.800 | .......................*****X | 8759.800 | | 7890.100 | .....................********.....*X | 9802.600 | | 9798.800 | ..........................*****X | 9798.800 | | 8179.100 | ......................**X | 8179.100 | | 7800.200 | .....................*****...***.*****X | 9407.000 | | 7678.100 | ....................*X | 7678.100 | | 7152.500 | ...................********.******...*X | 8715.700 | | 8451.800 | ......................*****X | 8451.800 | | 7646.000 | ....................*******X | 7666.400 | | 6225.700 | ................**X | 6225.700 | | 5780.300 | ...............********...******....................**..*****X | 14849.000 | | 12321.000 | .................................**.******..........*******X | 16245.100 | | 13016.000 | ...................................***.******..........*******X | 16714.800 | | 15435.000 | ..........................................*X | 15435.000 | | 15363.600 | ..........................................*****X | 15363.600 | | 13505.700 | .....................................*X | 13505.700 | | 13164.000 | ....................................*****X | 13164.000 | | 11281.000 | ..............................**X | 11281.000 | | 10708.600 | .............................******.....*******X | 12687.800 | | 11889.500 | ................................*******X | 11920.100 | | 10368.100 | ............................*X | 10368.100 | | 9565.000 | ..........................****X | 9565.000 | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 06:51:33,713 | test | INFO | pool-6-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------+----------+ | 9565.000 | ...................................................................................................X | 9565.000 | +----------+------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 06:51:33,857 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 23.6307927997 | 11.45 GiB | 8.46 GiB | 52.50 MiB / 3.50 GiB | 6513676 / 13082280 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 24.0311065783 | 11.45 GiB | 8.48 GiB | 24.25 MiB / 3.50 GiB | 6523079 / 13037583 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 25.8175779254 | 11.45 GiB | 8.38 GiB | 28.14 MiB / 3.50 GiB | 6556751 / 13027521 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 20.690533826 | 11.45 GiB | 8.51 GiB | 0.0 Byte / 3.50 GiB | 6541541 / 13092393 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 26.745667686 | 11.45 GiB | 8.34 GiB | 0.0 Byte / 3.50 GiB | 6502875 / 13030027 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 06:51:58,019 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 06:52:12,562 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 06:52:12,562 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 06:52:23,720 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 06:52:23,720 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 06:52:35,111 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 06:52:35,111 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 06:52:53,953 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 249.00 MiB | 304.90 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 492.85 MiB | 585.37 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.74 GiB | 48.55 GiB | 26.0476469685 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 06:52:53,954 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 06:52:58,430 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_recovery 2022-02-02 06:52:58,431 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_recovery 2022-02-02 06:52:58,431 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:670] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 06:53:19,747 | test | INFO | pool-6-thread-4 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 21.1789999008 seconds to complete 2022-02-02 06:53:19,753 | test | INFO | pool-6-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 06:58:24,756 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 06:58:24,757 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 06:58:24,765 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 07:33:33,351 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................**................................*****X | 434.000 | | 134.600 | ..............................**X | 134.600 | | 0.000 | ******....................**..........*****X | 130.500 | | 77.900 | ................X | 77.900 | | 77.900 | ................*X | 77.900 | | 52.500 | ...........****X | 52.500 | | 28.100 | .....**X | 28.100 | | 11.000 | .*****X | 11.000 | | 0.000 | ******************************************............***........*****X | 88.500 | | 61.700 | .............*X | 61.700 | | 27.500 | .....*****X | 27.500 | | 0.000 | *************************....*******X | 21.500 | | 10.500 | .**X | 10.500 | | 8.000 | ******..............**.....*****X | 90.000 | | 32.400 | ......*X | 32.400 | | 10.500 | .******.**X | 16.000 | | 13.500 | ..******.....**.....******.......*X | 90.000 | | 84.500 | ..................*****X | 84.500 | | 32.500 | ......**X | 32.500 | | 15.900 | ..******......**..*****X | 52.000 | | 22.100 | ....*X | 22.100 | | 10.000 | .******.*********..**..********X | 34.100 | | 27.000 | .....****X | 27.000 | | 10.500 | .**X | 10.500 | | 8.000 | ******.*X | 10.400 | | 10.000 | .*****X | 10.000 | | 7.600 | ***********.******.......*X | 39.800 | | 39.000 | .......*****X | 39.000 | | 6.700 | **X | 6.700 | | 0.000 | ******.....**..*****X | 34.000 | | 15.000 | ..********......**X | 39.500 | | 34.900 | .......****X | 34.900 | | 10.600 | .**X | 10.600 | | 10.500 | .*****X | 10.500 | | 10.400 | .*X | 10.400 | | 5.500 | ********.******.*******X | 16.000 | | 5.000 | **X | 5.000 | | 0.000 | ******.********...**.*****X | 26.000 | | 12.700 | .**X | 12.700 | | 8.000 | *****X | 8.000 | | 7.900 | **..******...**X | 28.500 | | 19.000 | ...****X | 19.000 | | 5.500 | ********X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | ***************..**..*****X | 18.500 | | 12.900 | .*X | 12.900 | | 5.500 | *****X | 5.500 | | 0.000 | *****************..**X | 10.000 | | 7.900 | ******..**..******..**.*****X | 36.000 | | 9.000 | .**X | 9.000 | | 0.000 | *****..***.******...**.*****X | 34.000 | | 9.500 | .*X | 9.500 | | 0.000 | ******************************************************************************************.....**..******..*******X | 42.000 | | 12.100 | .**X | 12.100 | | 4.600 | ******......*******X | 31.400 | | 12.500 | .**X | 12.500 | | 10.000 | .*****X | 10.000 | | 2.100 | *X | 2.100 | | 0.000 | ***************************************.....***.*****X | 30.000 | | 6.300 | *X | 6.300 | | 0.000 | **************..........***...*****X | 58.000 | | 12.900 | .*X | 12.900 | | 0.000 | ******.**.******...**X | 23.900 | | 23.500 | ....*****X | 23.500 | | 4.600 | *X | 4.600 | | 0.000 | ****************************************************************...**.*****X | 18.000 | | 4.500 | **X | 4.500 | | 0.000 | *******************************.**..******..*X | 25.600 | | 21.400 | ...*****X | 21.400 | | 8.000 | ********X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | ******....**..*****X | 28.000 | | 6.600 | **X | 6.600 | | 0.000 | **************.********...**..******..********....**.*****X | 56.700 | | 36.000 | .......**X | 36.000 | | 29.000 | .....*****X | 29.000 | | 15.600 | ..**.*******X | 20.900 | | 15.500 | ..******.********.********..********X | 34.000 | | 17.200 | ..*X | 17.200 | | 12.400 | .******.......**..*****X | 49.600 | | 25.000 | ....**X | 25.000 | | 18.000 | ...********..*****.*..********..*X | 49.300 | | 40.500 | ........*****X | 40.500 | | 15.900 | ..**.******..**X | 26.500 | | 19.200 | ...****X | 19.200 | | 16.500 | ..X | 16.500 | | 10.400 | .********......**.******..**...*****X | 62.800 | | 40.700 | ........**X | 40.700 | | 23.600 | ....*****X | 23.600 | | 10.100 | .*X | 10.100 | | 5.600 | *****X | 5.600 | | 0.000 | ****************......***..*****X | 38.100 | | 24.600 | ....*X | 24.600 | | 18.000 | ...******.*******X | 25.900 | | 4.600 | **X | 4.600 | | 0.000 | ******..**..*****X | 18.400 | | 12.800 | .*X | 12.800 | | 5.600 | *****X | 5.600 | | 0.000 | ***.*****...**X | 18.000 | | 13.500 | ..******...**..*****X | 38.000 | | 8.700 | .*X | 8.700 | | 0.000 | *******************************....**.*****X | 26.000 | | 14.900 | ..**X | 14.900 | | 10.000 | .****X | 10.000 | | 2.100 | **X | 2.100 | | 0.000 | ******...**.******.********...*******X | 35.200 | | 28.600 | .....**X | 28.600 | | 28.000 | .....****X | 28.000 | | 6.600 | **X | 6.600 | | 0.000 | **************.**.*****X | 10.000 | | 2.100 | ********............***....*****X | 70.300 | | 29.800 | .....*X | 29.800 | | 13.600 | ..******.**.*****X | 26.000 | | 6.900 | *X | 6.900 | | 0.000 | ******.*********.....**..******.*X | 42.200 | | 35.200 | .......*****X | 35.200 | | 7.200 | **X | 7.200 | | 2.100 | *****..*********.....**..*****X | 39.600 | | 9.300 | .*X | 9.300 | | 0.000 | ******.*********.********.*X | 15.700 | | 13.600 | ..*****X | 13.600 | | 2.400 | **X | 2.400 | | 0.000 | **************.....**..*****X | 34.000 | | 9.300 | .*X | 9.300 | | 2.100 | ******..**X | 10.000 | | 7.900 | *****X | 7.900 | | 0.000 | ********.**.*****X | 10.000 | | 2.100 | **X | 2.100 | | 0.000 | **********************...***..****X | 24.000 | | 7.200 | **X | 7.200 | | 0.000 | ******.*******X | 8.000 | | 2.400 | *X | 2.400 | | 0.000 | ******.***.*****X | 10.000 | | 2.100 | *X | 2.100 | | 0.000 | ***************************************.*******X | 8.000 | | 2.400 | **X | 2.400 | | 0.000 | **************.**..******....**X | 32.000 | | 24.800 | ....*****X | 24.800 | | 2.400 | *X | 2.400 | | 0.000 | ******.********..**.*****X | 18.400 | | 12.800 | .**X | 12.800 | | 5.600 | *****X | 5.600 | | 0.000 | *************************************************...***.****X | 18.000 | | 4.500 | **X | 4.500 | | 2.400 | ******.*X | 8.000 | | 5.600 | *****X | 5.600 | | 0.000 | *******X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 07:33:34,513 | test | INFO | pool-6-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................**................................*****X | 184.000 | | 57.500 | ..............................**X | 57.500 | | 0.000 | ******.....................................**................*****X | 98.000 | | 57.100 | ..............................*X | 57.100 | | 40.000 | ....................*****X | 40.000 | | 29.000 | ..............********X | 29.000 | | 23.500 | ...........*X | 23.500 | | 11.000 | ....*****X | 11.000 | | 0.000 | *****************.....**...*****X | 16.000 | | 5.000 | .*X | 5.000 | | 0.000 | ******.................***..........*****X | 50.500 | | 28.500 | ..............*X | 28.500 | | 13.500 | ......*****X | 13.500 | | 2.500 | **X | 2.500 | | 0.000 | ***************************************.....***...*****X | 16.000 | | 10.500 | ....*X | 10.500 | | 8.000 | ...*****X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | ******.....**..........******.........................**X | 74.900 | | 72.100 | ......................................*****X | 72.100 | | 25.600 | ............*X | 25.600 | | 13.500 | ......******...................***...........*****X | 68.500 | | 27.600 | ..............*X | 27.600 | | 5.500 | .*****X | 5.500 | | 0.000 | ********........***...........******......*X | 47.500 | | 32.500 | ................*****X | 32.500 | | 27.000 | .............*******X | 27.000 | | 16.000 | .......********X | 16.000 | | 5.000 | .*X | 5.000 | | 0.000 | ******.................**........*****X | 46.500 | | 25.100 | ............**X | 25.100 | | 13.500 | ......********.*************X | 16.000 | | 10.500 | ....**X | 10.500 | | 8.000 | ...******...**....******...*X | 26.500 | | 24.000 | ............*****X | 24.000 | | 18.500 | .........**X | 18.500 | | 11.000 | ....******...........**......******.........**....*****X | 66.000 | | 49.400 | .........................**X | 49.400 | | 42.000 | .....................****X | 42.000 | | 23.100 | ...........**X | 23.100 | | 21.000 | ..........******...*X | 26.500 | | 19.000 | .........*****X | 19.000 | | 2.500 | **X | 2.500 | | 0.000 | ********.******.......*X | 15.900 | | 15.500 | .......*****X | 15.500 | | 2.100 | **X | 2.100 | | 0.000 | ********..******......*X | 16.000 | | 11.000 | ....*****X | 11.000 | | 0.000 | ***************************************************....**.******....***.*****.***..*****X | 24.000 | | 13.000 | ......*X | 13.000 | | 8.000 | ...*****X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | *************************************************************************..............**......*****X | 38.400 | | 17.200 | ........**X | 17.200 | | 5.600 | ..******......................................**...............*****X | 104.000 | | 39.100 | ....................**X | 39.100 | | 16.000 | .......****X | 16.000 | | 5.000 | .**X | 5.000 | | 0.000 | **********************.*.**..*****X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | *******************************.............***.....*****X | 34.000 | | 9.600 | ....*X | 9.600 | | 0.000 | **************.............***.....*****X | 34.000 | | 15.000 | .......*X | 15.000 | | 8.000 | ...******.**.*****X | 12.500 | | 10.100 | ....**X | 10.100 | | 5.500 | .*****X | 5.500 | | 0.000 | ******************************************************************..**..*****X | 8.000 | | 2.500 | **X | 2.500 | | 0.000 | ******..**..*****X | 8.000 | | 2.500 | *X | 2.500 | | 0.000 | ***************....**.******..............**......*****X | 46.000 | | 11.300 | .....**X | 11.300 | | 0.000 | **************.....**...*****X | 16.000 | | 5.000 | .**X | 5.000 | | 0.000 | ****************..******............................**.......******..................**........*****X | 116.700 | | 52.200 | ...........................**X | 52.200 | | 21.600 | ..........****X | 21.600 | | 19.000 | .........***.*****X | 21.500 | | 5.000 | .*X | 5.000 | | 0.000 | ********.******....**X | 10.000 | | 7.900 | ...******....................**.........*****X | 62.000 | | 59.600 | ...............................*******X | 60.000 | | 16.500 | .......*X | 16.500 | | 0.000 | ******.....................................**.............*****X | 92.000 | | 22.600 | ...........**X | 22.600 | | 0.000 | ******.....**......******............********.......**.....*****X | 64.500 | | 40.600 | .....................**X | 40.600 | | 36.100 | ..................****X | 36.100 | | 35.200 | ..................**X | 35.400 | | 21.400 | ..........*****X | 21.400 | | 13.400 | ......**..******.........***....*****............***....*****X | 70.000 | | 18.600 | .........*X | 18.600 | | 4.500 | .******..................**.*****X | 39.500 | | 7.000 | ..**X | 7.000 | | 0.000 | ******..........**....*****X | 26.000 | | 22.900 | ...........*X | 22.900 | | 22.100 | ...........*****X | 22.100 | | 14.200 | ......**X | 14.200 | | 7.900 | ...****X | 7.900 | | 0.000 | *********.......**..*****X | 18.000 | | 10.100 | ....**.******........**X | 26.000 | | 21.500 | ..........*****X | 21.500 | | 2.400 | *X | 2.400 | | 0.000 | ******....**..******............................***........*****X | 77.600 | | 18.100 | ........*X | 18.100 | | 0.000 | ********..******..........**X | 23.600 | | 21.400 | ..........*****X | 21.400 | | 2.400 | *X | 2.400 | | 0.000 | *****************.*****..........***.*****X | 23.900 | | 18.300 | ........*X | 18.300 | | 18.000 | ........*****X | 18.000 | | 4.500 | .**....******......................*X | 51.600 | | 42.900 | ......................*****X | 42.900 | | 27.100 | .............**....*****X | 34.000 | | 9.300 | ....**X | 9.300 | | 4.500 | .******.......*X | 18.000 | | 13.500 | ......*****X | 13.500 | | 5.600 | ..**..*******X | 10.400 | | 5.600 | ..******.................***.........****X | 54.100 | | 31.700 | ................**X | 31.700 | | 17.900 | ........******........**...*****X | 38.000 | | 27.800 | ..............*X | 27.800 | | 26.000 | .............*****X | 26.000 | | 20.400 | ..........**X | 20.400 | | 18.000 | ........****X | 18.000 | | 12.400 | .....***.******.....*X | 22.100 | | 15.800 | .......*****X | 15.800 | | 5.600 | ..**.*****X | 8.000 | | 2.400 | ***.******......................**..*****X | 49.500 | | 9.000 | ...*X | 9.000 | | 0.000 | ******....***..******...*******X | 18.300 | | 10.400 | ....*X | 10.400 | | 8.000 | ...******......***..****X | 23.700 | | 20.400 | ..........**X | 20.400 | | 15.900 | .......******......**....*****X | 34.000 | | 17.200 | ........*X | 17.200 | | 14.500 | ......******...**X | 20.100 | | 13.500 | ......*****X | 13.500 | | 0.000 | ****************....***.****X | 10.000 | | 2.100 | **X | 2.100 | | 0.000 | **************..........**....*****X | 26.000 | | 6.900 | ..**X | 6.900 | | 0.000 | ******......**..*****X | 16.000 | | 10.400 | ....*X | 10.400 | | 8.000 | ...**************X | 8.000 | | 2.400 | *X | 2.400 | | 0.000 | ********..******......**X | 16.000 | | 11.200 | .....*****X | 11.200 | | 0.000 | ***************************.******...*X | 8.000 | | 5.600 | ..********...******.......**X | 24.000 | | 19.200 | .........*****X | 19.200 | | 2.400 | *X | 2.400 | | 0.000 | ********..******.........**...******......**X | 37.600 | | 32.800 | ................*****X | 32.800 | | 4.800 | .*X | 4.800 | | 2.400 | ******...*X | 8.000 | | 5.600 | ..*****X | 5.600 | | 0.000 | ********...***.*****X | 8.000 | | 2.400 | *X | 2.400 | | 0.000 | ******.........**....*****X | 24.000 | | 18.400 | ........**X | 18.400 | | 16.000 | .......****X | 16.000 | | 4.800 | .**X | 4.800 | | 0.000 | **********************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 07:33:35,203 | test | INFO | pool-6-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.....**..******.................................................................***.......................*****X | 30329.600 | | 26190.900 | .................................................................................*X | 26190.900 | | 23058.400 | .......................................................................*****X | 23058.400 | | 8507.800 | .........................*X | 8507.800 | | 4530.100 | .............******...............***......*****X | 11457.700 | | 6888.000 | ....................*X | 6888.000 | | 5146.000 | ...............******.....**...******....................***.....*****X | 15677.400 | | 11902.400 | ....................................*X | 11902.400 | | 10702.300 | ................................*****X | 10702.300 | | 6623.300 | ...................*X | 6623.300 | | 5249.300 | ...............*****X | 5249.300 | | 4873.400 | ..............***.******........**..*****X | 8467.000 | | 7508.300 | ......................*X | 7508.300 | | 7351.600 | ......................*****X | 7351.600 | | 4580.400 | .............**X | 4580.400 | | 3591.400 | ..........******....**...******.........**...******.**X | 9874.700 | | 9626.900 | .............................******...............**.......******......**.*****X | 18900.100 | | 14964.900 | .............................................**..******...................................**......*****X | 28458.300 | | 16027.200 | .................................................**X | 16027.200 | | 11270.600 | ..................................*****X | 11270.600 | | 8050.800 | ........................*X | 8050.800 | | 7954.600 | .......................*X | 7954.600 | | 7954.600 | .......................*X | 7954.600 | | 7954.600 | .......................**............**.......*****X | 13749.900 | | 12272.500 | .....................................**X | 12272.500 | | 11218.400 | ..................................*****X | 11218.400 | | 10246.500 | ...............................*X | 10246.500 | | 9799.500 | .............................******.....**...******.**X | 12466.500 | | 12333.700 | .....................................*****..*********....**...*****X | 15051.300 | | 12663.100 | ......................................*X | 12663.100 | | 11281.200 | ..................................******..........***..*****X | 15106.400 | | 11557.200 | ...................................*X | 11557.200 | | 11098.000 | .................................******..**..******..............***.....*****..................***..*****X | 24845.800 | | 16990.900 | ....................................................*X | 16990.900 | | 14911.800 | .............................................******..............**....*****X | 20496.700 | | 16042.900 | .................................................*X | 16042.900 | | 14113.300 | ...........................................******.....***......******...........**.******...........**.....*****X | 26611.100 | | 21499.200 | ..................................................................*X | 21499.200 | | 19461.000 | ............................................................*****X | 19461.000 | | 18033.500 | .......................................................**X | 18033.500 | | 16809.300 | ...................................................*****X | 16809.300 | | 11448.500 | ..................................*X | 11448.500 | | 11158.000 | ..................................*****X | 11158.000 | | 9859.400 | .............................*X | 9859.400 | | 8712.900 | ..........................*****X | 8712.900 | | 7635.200 | ......................***.*****.....***.*****X | 9848.500 | | 8923.400 | ...........................*X | 8923.400 | | 8285.000 | .........................*****X | 8285.000 | | 4687.000 | .............X | 4687.000 | | 4687.000 | .............X | 4687.000 | | 3652.600 | ..........******...........***...****X | 8025.300 | | 3920.300 | ...........**X | 3920.300 | | 2534.100 | ......******.**..******.**..X | 4154.300 | | 4154.300 | ............*****.........*X | 7193.800 | | 7193.800 | .....................X | 7193.800 | | 6800.300 | ....................*X | 6800.300 | | 6800.300 | ....................**X | 6800.300 | | 6800.300 | ....................X | 6800.300 | | 2967.300 | ........*X | 2967.300 | | 2565.100 | .......******.......**.***X | 5368.700 | | 5368.700 | ...............*X | 5368.700 | | 2952.000 | ........**X | 2952.000 | | 2101.300 | .....*****....***...**X | 4323.900 | | 4323.900 | ............***..............**..*****X | 9535.700 | | 4269.200 | ............X | 4269.200 | | 4269.200 | ............X | 4269.200 | | 2568.000 | .......*****X | 2568.000 | | 822.600 | .*X | 822.600 | | 822.600 | .X | 822.600 | | 616.700 | ******...**..******..........**.*****X | 5580.500 | | 1839.300 | ....**X | 1839.300 | | 1244.500 | ..*****...*.....**..******....**..*****X | 6302.300 | | 2054.400 | .....**X | 2054.400 | | 940.300 | .*****................***....******....**..*****X | 9097.600 | | 3919.000 | ...........**X | 3919.000 | | 1748.600 | ....*****............X | 5499.800 | | 5499.800 | ................**......******...........**.*****X | 11310.600 | | 4341.200 | ............*X | 4341.200 | | 2585.400 | .......*X | 2585.400 | | 2585.400 | .......****............***.....******.*X | 8358.900 | | 7131.400 | .....................*****X | 7131.400 | | 2442.100 | ......*X | 2442.100 | | 1862.200 | ....******..********X | 2321.600 | | 1913.800 | .....*X | 1913.800 | | 1752.700 | ....**************....***..******.....**.*****X | 5667.100 | | 2397.300 | ......*X | 2397.300 | | 1431.000 | ...******.***...******.......................**...*****X | 10938.500 | | 4363.000 | ............*X | 4363.000 | | 3095.800 | ........******...............***........*****X | 10198.300 | | 7708.100 | .......................*X | 7708.100 | | 5860.600 | .................*****X | 5860.600 | | 2153.300 | .....*X | 2153.300 | | 1546.300 | ...******.**X | 1651.700 | | 1379.600 | ...******...................**......*****X | 9537.200 | | 3791.000 | ..........*X | 3791.000 | | 2011.000 | .....******..***......*****...........................**X | 13080.600 | | 12832.500 | .......................................*****X | 12832.500 | | 3821.600 | ..........*X | 3821.600 | | 2096.600 | .....******.***...***X | 3212.700 | | 3212.700 | .........*.........................***.....*****X | 12998.000 | | 4887.200 | ..............*X | 4887.200 | | 1888.100 | ....******...........**.......*****X | 7599.900 | | 6980.300 | ....................**X | 6980.300 | | 6549.800 | ...................******.*X | 6908.700 | | 6726.600 | ....................*****X | 6726.600 | | 5519.400 | ................**..******...........**X | 9566.600 | | 9067.200 | ...........................*****X | 9067.200 | | 4655.600 | .............*X | 4655.600 | | 4043.500 | ...........******............**..******...***..*****X | 9966.100 | | 4987.000 | ..............*X | 4987.000 | | 4245.400 | ............**X | 4245.400 | | 4245.400 | ............*X | 4245.400 | | 4245.400 | ............*.........*X | 7012.600 | | 6770.900 | ....................******..................................**...................******..........................*X | 31853.800 | | 31028.000 | ................................................................................................*****X | 31028.000 | | 25427.800 | ..............................................................................**X | 25427.800 | | 23920.500 | ..........................................................................*****X | 23920.500 | | 18940.800 | ..........................................................*X | 18940.800 | | 18235.600 | ........................................................******...........*X | 21826.400 | | 21345.300 | ..................................................................*****X | 21345.300 | | 10663.500 | ................................*X | 10663.500 | | 7912.300 | .......................******...........***........*****..................***..*****X | 20321.200 | | 18411.700 | ........................................................*X | 18411.700 | | 17979.900 | .......................................................******...............**....*****X | 23912.000 | | 17020.600 | ....................................................*X | 17020.600 | | 15349.900 | ...............................................******.......................**.......*****X | 24877.500 | | 19505.300 | ............................................................**X | 19505.300 | | 17195.700 | ....................................................****X | 17195.700 | | 16436.800 | ..................................................***.....*****..........*......**...*****X | 23985.900 | | 23636.500 | .........................................................................*X | 23636.500 | | 21768.800 | ...................................................................*****X | 21768.800 | | 17668.500 | ......................................................**..*****X | 18312.200 | | 17864.600 | .......................................................*X | 17864.600 | | 16455.900 | ..................................................*****X | 16455.900 | | 15198.800 | ..............................................***....*****...................***..*****X | 23088.600 | | 20657.900 | ...............................................................*X | 20657.900 | | 19216.800 | ...........................................................*****X | 19216.800 | | 8288.700 | .........................*X | 8288.700 | | 5724.000 | ................******...........***.....*****..........**X | 13746.700 | | 13638.200 | .........................................*****X | 13638.200 | | 10659.000 | ................................*X | 10659.000 | | 9945.400 | ..............................******..**...******....*X | 12819.700 | | 12393.400 | .....................................*****X | 12393.400 | | 11311.500 | ..................................**X | 11311.500 | | 10705.500 | ................................*****X | 10705.500 | | 10495.600 | ...............................**..******.*X | 11183.300 | | 11064.200 | .................................******.....**..******.....***..****X | 15569.200 | | 15006.700 | ..............................................**X | 15006.700 | | 14309.600 | ...........................................*****X | 14309.600 | | 11879.500 | ....................................*X | 11879.500 | | 10943.000 | .................................*****X | 10943.000 | | 7725.300 | .......................*X | 7725.300 | | 6521.000 | ...................*****X | 6521.000 | | 6429.200 | ...................**...******.........*******X | 10481.000 | | 7728.300 | .......................**X | 7728.300 | | 7342.900 | ......................******.........**...******..............**..*****X | 16466.700 | | 15313.000 | ...............................................**X | 15313.000 | | 15073.300 | ..............................................*****.........***...*****X | 19001.900 | | 17213.300 | .....................................................*X | 17213.300 | | 16470.200 | ..................................................******....**.******..**.*****X | 18988.800 | | 17665.800 | ......................................................***.****X | 18045.400 | | 17475.800 | .....................................................**X | 17475.800 | | 15732.500 | ................................................*****X | 15732.500 | | 9980.100 | ..............................*X | 9980.100 | | 9219.900 | ...........................******...**.******.................***...****X | 16778.200 | | 15287.600 | ..............................................***..*******X | 15794.100 | | 14639.400 | ............................................******.**...******....*X | 16944.000 | | 16802.300 | ...................................................*****X | 16802.300 | | 15750.600 | ................................................**X | 15750.600 | | 15714.700 | ................................................*****.............***....*****X | 21301.900 | | 18442.200 | ........................................................*X | 18442.200 | | 17475.900 | .....................................................*****X | 17475.900 | | 14784.400 | .............................................*X | 14784.400 | | 13462.100 | .........................................*****X | 13462.100 | | 11093.400 | .................................***.****X | 11320.300 | | 10395.800 | ...............................**X | 10395.800 | | 9572.600 | .............................******...........**....******.*X | 14793.900 | | 14533.200 | ............................................*****X | 14533.200 | | 9589.300 | .............................**X | 9589.300 | | 9317.000 | ............................******................**.*****X | 14851.400 | | 14164.700 | ...........................................*X | 14164.700 | | 14091.900 | ...........................................*****X | 14091.900 | | 8161.000 | ........................*X | 8161.000 | | 6527.100 | ...................******..***.******.......*X | 9778.800 | | 9630.500 | .............................*****X | 9630.500 | | 4709.900 | .............*X | 4709.900 | | 3886.100 | ...........******.....***.*****X | 5857.600 | | 4757.900 | .............*X | 4757.900 | | 4217.300 | ............*****X | 4217.300 | | 2885.300 | ........********.........***...******.........**...*****X | 10629.200 | | 7158.500 | .....................*X | 7158.500 | | 5712.300 | ................*****X | 5712.300 | | 4892.300 | ..............***.******..*X | 5842.200 | | 5586.100 | ................******..*X | 6111.900 | | 5968.900 | .................*****X | 5968.900 | | 3931.700 | ...........**X | 3931.700 | | 3764.800 | ..........****X | 3764.800 | | 3700.900 | ..........**X | 3700.900 | | 3606.900 | ..........**************...********.....***.*****X | 6486.700 | | 4551.800 | .............*X | 4551.800 | | 3956.700 | ...........******..**.*****X | 4851.700 | | 4484.900 | .............*********.........**...******......*******X | 10401.000 | | 9665.600 | .............................********.........***...*****X | 13414.100 | | 9721.900 | .............................*X | 9721.900 | | 8774.800 | ..........................******..*X | 9256.400 | | 9087.800 | ...........................*****X | 9087.800 | | 8276.500 | ........................**X | 8276.500 | | 8100.100 | ........................****X | 8100.100 | | 7922.200 | .......................**X | 7922.200 | | 7780.100 | .......................******.**...******...............*******X | 13941.100 | | 7656.200 | .......................**X | 7656.200 | | 6374.500 | ...................****X | 6374.500 | | 6321.000 | ..................***.******..........**...******.....**...*****X | 13132.900 | | 11743.800 | ...................................**X | 11743.800 | | 11152.200 | ..................................*****.......********X | 13596.600 | | 7716.800 | .......................X | 7716.800 | +-----------+---------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 07:33:36,502 | test | INFO | pool-6-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------+----------+ | 7716.800 | ...................................................................................................X | 7716.800 | +----------+------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 07:33:36,677 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 07:33:36,703 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 07:34:06,316 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 07:34:06,325 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 07:34:06,418 | test | INFO | pool-6-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 7.90135883241 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 4.68357609242 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 6.32847815168 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.338303470743 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 4.67184368737 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 07:34:06,436 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 07:34:16,500 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 07:34:26,565 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 17.96 2022-02-02 07:34:36,601 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 33.45 2022-02-02 07:34:46,647 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 38.03 2022-02-02 07:34:56,693 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 49.75 2022-02-02 07:35:06,729 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 56.19 2022-02-02 07:35:16,766 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 63.0 2022-02-02 07:35:26,815 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 66.67 2022-02-02 07:35:36,849 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 67.02 2022-02-02 07:35:46,884 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 67.37 2022-02-02 07:35:56,918 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 67.37 2022-02-02 07:36:06,983 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 67.61 2022-02-02 07:36:17,016 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 67.96 2022-02-02 07:36:27,062 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.02 2022-02-02 07:36:37,114 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.19 2022-02-02 07:36:47,151 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.54 2022-02-02 07:36:57,184 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.6 2022-02-02 07:37:07,216 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.84 2022-02-02 07:37:17,247 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 69.19 2022-02-02 07:37:27,282 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 69.6 2022-02-02 07:37:37,316 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 69.89 2022-02-02 07:37:47,351 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:37:57,381 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:38:07,420 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:38:17,453 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:38:27,509 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:38:37,542 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:38:47,576 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:38:57,609 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:39:07,644 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:39:17,683 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:39:27,714 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:39:37,746 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:39:47,783 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:39:57,819 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:40:07,851 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:40:17,881 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:40:27,915 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:40:37,960 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:40:47,990 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:40:58,023 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:41:08,055 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:41:18,088 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.13 2022-02-02 07:41:28,125 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.47 2022-02-02 07:41:38,154 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.52 2022-02-02 07:41:48,184 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.58 2022-02-02 07:41:58,213 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.58 2022-02-02 07:42:08,247 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.73 2022-02-02 07:42:18,286 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.84 2022-02-02 07:42:28,318 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 71.19 2022-02-02 07:42:38,349 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 71.48 2022-02-02 07:42:48,390 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 71.72 2022-02-02 07:42:58,438 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 71.84 2022-02-02 07:43:08,471 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.19 2022-02-02 07:43:18,507 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.43 2022-02-02 07:43:28,539 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.61 2022-02-02 07:43:38,582 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.66 2022-02-02 07:43:48,618 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.66 2022-02-02 07:43:58,661 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.66 2022-02-02 07:44:08,703 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.72 2022-02-02 07:44:18,737 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.83 2022-02-02 07:44:28,785 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.89 2022-02-02 07:44:38,816 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 73.18 2022-02-02 07:44:48,848 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 73.52 2022-02-02 07:44:58,878 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 73.87 2022-02-02 07:45:08,914 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 74.04 2022-02-02 07:45:18,957 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 74.22 2022-02-02 07:45:28,990 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 74.5 2022-02-02 07:45:39,032 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 74.84 2022-02-02 07:45:49,062 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 75.1 2022-02-02 07:45:59,104 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 75.37 2022-02-02 07:46:09,138 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 75.77 2022-02-02 07:46:19,176 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 76.41 2022-02-02 07:46:29,213 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 76.7 2022-02-02 07:46:39,260 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 76.98 2022-02-02 07:46:49,296 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 77.44 2022-02-02 07:46:59,332 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 77.67 2022-02-02 07:47:09,362 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 77.72 2022-02-02 07:47:19,398 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 78.01 2022-02-02 07:47:29,433 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 78.17 2022-02-02 07:47:39,471 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 78.39 2022-02-02 07:47:49,509 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 79.01 2022-02-02 07:47:59,546 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 79.34 2022-02-02 07:48:09,578 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 79.82 2022-02-02 07:48:19,611 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 80.3 2022-02-02 07:48:29,653 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 81.04 2022-02-02 07:48:39,696 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 81.31 2022-02-02 07:48:49,730 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 81.68 2022-02-02 07:48:59,762 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.0 2022-02-02 07:49:09,811 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.32 2022-02-02 07:49:19,845 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.52 2022-02-02 07:49:29,884 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.63 2022-02-02 07:49:39,918 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.74 2022-02-02 07:49:49,951 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.95 2022-02-02 07:49:59,987 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.06 2022-02-02 07:50:10,026 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.22 2022-02-02 07:50:20,061 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.37 2022-02-02 07:50:30,091 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.42 2022-02-02 07:50:40,128 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.7 2022-02-02 07:50:50,161 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 84.01 2022-02-02 07:51:00,194 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 84.48 2022-02-02 07:51:10,230 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 85.04 2022-02-02 07:51:20,266 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 85.25 2022-02-02 07:51:30,299 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 85.62 2022-02-02 07:51:40,332 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 85.89 2022-02-02 07:51:50,368 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.15 2022-02-02 07:52:00,401 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.47 2022-02-02 07:52:10,431 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.52 2022-02-02 07:52:20,463 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.73 2022-02-02 07:52:30,496 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.78 2022-02-02 07:52:40,556 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.0 2022-02-02 07:52:50,592 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.1 2022-02-02 07:53:00,627 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.2 2022-02-02 07:53:10,661 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.36 2022-02-02 07:53:20,690 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.62 2022-02-02 07:53:30,720 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.67 2022-02-02 07:53:40,755 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.09 2022-02-02 07:53:50,792 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.19 2022-02-02 07:54:00,826 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.55 2022-02-02 07:54:10,858 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.76 2022-02-02 07:54:20,891 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.87 2022-02-02 07:54:30,938 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.02 2022-02-02 07:54:40,971 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.23 2022-02-02 07:54:51,006 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.28 2022-02-02 07:55:01,038 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.49 2022-02-02 07:55:11,068 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.7 2022-02-02 07:55:21,098 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.75 2022-02-02 07:55:31,130 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.8 2022-02-02 07:55:41,163 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 90.12 2022-02-02 07:55:51,201 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 90.32 2022-02-02 07:55:51,539 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 81 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_gYtdRKkEHiGb0N0b-32-332000_kcXWiIKiodtIk9LrNL1%IFsZNL2HvQBZg4l0ZIFZS7VqY77Q2ghcPye5WF9tWG-AiDmpNHvscHVoBkY9Rk7npE%sR0ddBU1gr_9H31Omzl2bi-s6PMIJRoZ3K-MeJgMG2TKOEaDrfR-RdKlVRZONL_%bGiVyNVRzWM4MBPWTepvV5lgeMB9fzXcj-ov57vPXm-DsuuWTM%qMJ-qb3qQXkOnpVJh5Fj-_zPZzxd4sgvRjjYlO-32-332000_ttl=0_delete__26060_51077_1643816051.13 2022-02-02 07:55:51,540 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 35 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-32-336000_Js-Pp_PJHet1kgSKlPQESgkuzjSESnMvJV7YUHGFB4wYBdcu8Xlj36_sj3w5zWPM%0jIOdC7I0bZyf9cU4vgKwNz3FQQYs3aw78k3gXqI2PsCl1AtqFBNJC-wYSAkefF20SrXWCG3gMex3QcjXmEDe3b_6_Tk2zx3cIYjIBsZvKCXa3Kdp6am61fY2PgrWjO3AnenFyb4J2yU-32-338000_ttl=0_read__51077_71091_1643816051.13 2022-02-02 07:55:51,540 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 44 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_gYtdRKkEHiGb0N0b-32-332000_08Kq1K%%WhoVLUl1dqIWpUORWGCoE0zMdPoBzqdpYbBwZWpdWBOyZtFnavT4GwsmCZEAD6_dNOusfq3KxNXxGO19AoUgewtaVX%V2_bOT_zTUiJ_pQB58_7hw4qjmSlT%xMDJCYZ404s7HQp3WGxWP6ibvpZw5-32-335000_ttl=0_delete__26060_51077_1643816051.13 2022-02-02 07:55:51,542 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 75 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_gYtdRKkEHiGb0N0b-32-332000_Bo5bm5ASl25XFpwS9Txsxy9d6Ddup%HDiwqphhN83iv5NV9ZE4Fsc_9T8q6LE8RTUZ_i51IlHowX2qeLuclqvtYQILDXXTJHxOA3lnEMDh1ki5-HPH9354FyA36I6k58fO2rNlhQjbTThfg0nc3MvCW1oW4szHxQos4UYgfe38QhniI576wSsbPKnZgLNUy1ncBfE46ksFc_2PowPtgo3_PsgEI2ork-AW0-32-333000_ttl=0_read__51077_71091_1643816051.14 2022-02-02 07:55:51,542 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 14 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-32-342000_ClUUnu6lkK1AvFvwhGTMjw_LwQLgzqo5MmComdzkDsswtSikh09p0DoGaTsUM5wkvlNbTiVnRiG_4MCEV%BDKblWj1%CzxeM8jn_VgXP732a5Z9moJhAqr5LTNL8ewRs6keo_uFb7gRytxUnfXJs9_-KW6V5dIuSihvm_ce4o34DzKyLDGGmvUcw9CJLyCy0s6z-32-344000_ttl=0_read__51077_71091_1643816051.14 2022-02-02 07:55:51,542 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 45 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-32-339000_1jEe-32-341000_ttl=0_read__51077_71091_1643816051.14 2022-02-02 07:55:56,309 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 37 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-32-336000_gzfeUopt_YwV-b5SXXBGOoJNh4ggVdWjBOZ-32-337000_ttl=0_read__51077_71091_1643816051.14 2022-02-02 07:55:58,948 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 45 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-32-336000_c-32-337000_ttl=0_read__51077_71091_1643816051.14 2022-02-02 07:56:01,240 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 90.58 2022-02-02 07:56:11,279 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.0 2022-02-02 07:56:11,740 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 37 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-32-322000_ZFe18IkMKsgH9QZOPZC1W1PKeICSizfISvkFzJl34PRTs_xnj3g%atLME9c4MjvTwfEsEhHJXwJCYyUK7G6NpRDF-CmojH1LJ%gh%IdszZgWXSkoR7K2j95bc-a0XatLQp34wM2RNYFQWT4-gZ-SV%S%oZHxxZ4lFRvZe0s1gN-32-323000_ttl=0_read__51077_71091_1643816051.14 2022-02-02 07:56:11,742 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 115 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-32-322000_HcV1pHvsl-32-324000_ttl=0_delete__26060_51077_1643816051.14 2022-02-02 07:56:21,214 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 34 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-32-336000_0PkpIeYZa%Ln_yGjGE0UM39cX1RP4ixk5hqY8br-fUtwAlI5XF1m3g1zwerg7xdB5%oxA0OBTJmF_dKEicmK0T%w2Js7d3yVlHZ0-32-338000_ttl=0_read__51077_71091_1643816051.15 2022-02-02 07:56:21,319 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.26 2022-02-02 07:56:27,190 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 36 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-32-327000_NSkct%HI4oait-HTg2k787QFL9bUtech0GbwKnQxo00y179d7KA_x31rYo-sxscLW4Y7n56QM_-JXLrXVSSYpi-l22wiZRpKxEgyKwQRkWr3iT%eWKzpJNSTjidol70Deyr4u%cla93DoMMFKuDK0DLS94fjI67NKxln9u3Y-fDr46XTauGIIFT_ttG1XIbA9NItnI8T_ENYqK-32-328000_ttl=0_read__51077_71091_1643816051.15 2022-02-02 07:56:31,355 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.62 2022-02-02 07:56:41,390 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.93 2022-02-02 07:56:51,424 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.93 2022-02-02 07:57:01,457 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 92.19 2022-02-02 07:57:11,487 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 92.4 2022-02-02 07:57:21,523 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 92.55 2022-02-02 07:57:26,562 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 121 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-32-339000_-PNjTw-rHWn_92yoZUzXK1FMjGwMQJNqgSacDrKkzHB4Dtvp3i9SzAyIibo_Ezeoh2JIevTC7ugqdoagj3xUr_eC2fxj7PoMavYJ7SjJf5fntNBGsKjWDbai955ONyWNz0nF-32-341000_ttl=0_delete__26060_51077_1643816051.19 2022-02-02 07:57:26,562 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 81 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_gYtdRKkEHiGb0N0b-32-332000_aO5pBxcncBSS4D9_TeKu-rdsXAaPyBP%AN0BoH5TtuDZL2W9AQkOMT5bRVTJsmquxpUzE0qB3DLpfYmN7zVBQTTN7tjS%Xxj-32-333000_ttl=0_read__51077_71091_1643816051.19 2022-02-02 07:57:26,562 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 35 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-32-324000_aOi346RI7BGSZ4Y7n%HSw_XPB-66XVUD_4GypZ9AGRcLvJ44EIAL4VJtMkEXuPs3IaZ7i_rNrFs5EvgJJrj-8f8oLGCiXbCCfBfhv1Zea1lOTU-32-324000_ttl=0_read__51077_71091_1643816051.19 2022-02-02 07:57:28,085 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 80 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-32-339000_5bU4ViIM27JuOlO_vg-32-339000_ttl=0_read__51077_71091_1643816051.19 2022-02-02 07:57:31,562 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 92.66 2022-02-02 07:57:36,038 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 157 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-32-319000_SdeXkdL2RgrH_sc4iM4sV9FBTYBTcIWwfwEeZg5NRfEo93XmefAswHj98jY01NeNvU22kxHdqezSLqYy_2qzRf7kLeyCd-32-320000_ttl=0_read__51077_71091_1643816051.19 2022-02-02 07:57:41,601 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 92.91 2022-02-02 07:57:41,671 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 142 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-32-319000_5Qb1555D-QNUCfFoXQUufqSio8FmUL%RQjh9F%aVaRV8092xdN4Zd%zds9hWJ-R32q5Odx_glvpNIl0H7nvZ5-X5urWIlxjJol-32-320000_ttl=0_read__51077_71091_1643816051.19 2022-02-02 07:57:51,658 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.02 2022-02-02 07:58:01,701 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.07 2022-02-02 07:58:11,737 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.18 2022-02-02 07:58:21,769 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.33 2022-02-02 07:58:31,802 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.59 2022-02-02 07:58:36,546 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 433 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-32-319000_zFv4bW9aJUDCJmvUT-5SsU4gJXJYeDmf8no24e13Gz1NiHRZshXSr0pZWOwB5_zl2FLcDVfYhmGvOLU27yocn982pBt7Pg7BQuT7L2IaFSikL0cXnzcBuzbbLvGDZB0of-pzu-LoUse4qk9rgAU3-KUiI9XlA83lDohq3nmtA9vDLOGkDERs1gB3S-32-321000_ttl=0_delete__26060_51077_1643816051.2 2022-02-02 07:58:36,549 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 155 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-32-327000_rjjnoMhS9eUGdI-32-328000_ttl=0_read__51077_71091_1643816051.2 2022-02-02 07:58:36,549 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 260 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-32-342000_7JL0f_-WOOYd1FHPF%cGyjQ1%eZEdnmQcEJ2IzuJYO%AaGZJVTMgExBfIF7H6HiSBk%sphi5oFocHgdYFvGiASa_iqyqnI2NU_i22m-32-342000_ttl=0_read__51077_71091_1643816051.2 2022-02-02 07:58:36,551 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 322 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-32-329000_ZMlbC7PYTy1Sdn_8ufcVNImCXYwZMZx1YBD48b-AOnSQMffh6i7mPtg9TAVRKB1E01Qn6TijGuNruX6sUdb7BEQcRlwiJARJFZQY2aNteTYdzabxGvgGNKgZLL%YRdVRyPte7VCG1a3zGDgO4lyP2uzBG0AXw1naYuCORcT10tdDHVLFa%SQJJ6FB2PYlQD8LdtUbxvpI-32-331000_ttl=0_delete__26060_51077_1643816051.2 2022-02-02 07:58:37,345 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 270 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-32-327000_Y-qgZgTNkAKd8mwLvSi-QY1sobDZhyGFg4SKnamoGySkobary%JOXCR7ZI8eLSQb2damrOAF%ijAOmBkXBKhBh8gL1zY1IG1gin1-32-328000_ttl=0_delete__26060_51077_1643816051.2 2022-02-02 07:58:37,346 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 418 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-32-342000_Pbwv%oUOgkDFjmp2gH6oAM9UkPqrV0mn8GsXqQcbKk0HresdaeSGHVCckqBI8ziK5G-Y52QlDx8tSM5%aA%da29i-wK9rAekMTeVAEU5ZnXfrzWuKoBcXjvedPIELV%_muEO1%M5X0PKdDOf5c-32-344000_ttl=0_read__51077_71091_1643816051.21 2022-02-02 07:58:37,346 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 236 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default__default__default_ttl=0_read__51077_71091_1643816051.21 2022-02-02 07:58:41,835 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.9 2022-02-02 07:58:42,638 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 381 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_gYtdRKkEHiGb0N0b-32-332000_gxxxR2Px8gfVz1IMf1r3oHvrsupRh0qt3p5J2juPEZu_j9%Foz3zCHPuYWb7NJ7sysQQe2Zmgej7GL-oNAN2Q0tfq7NSgKPivZyAgTbZVzZTW21rLH3afS-32-333000_ttl=0_delete__26060_51077_1643816051.21 2022-02-02 07:58:42,638 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 141 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-32-324000_d-4d0W78bqxC8THAnvyAaVtz58X7iBjuZ3eRq1AHXeNH1yp7rfNpCh-32-324000_ttl=0_read__51077_71091_1643816051.21 2022-02-02 07:58:44,164 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 272 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-32-319000_6l7lw7dW8LpoFdepXN%JZOWN%GcxN2yUXy5%Cn4je_Stj16LuyCBjdKi4%pRQpu3KWgk2CBqiZnhosrcFRJ-dSa095iOpZ4y0PyWmKKWoLDEDQ9a7DC2pYZ1xMOtnDtjGQI-%5-32-321000_ttl=0_delete__26060_51077_1643816051.22 2022-02-02 07:58:44,164 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 197 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-32-339000_FKaO5a8%3e2ezgWbH3SvPznAIyatJJIIOO1xqisXOP2GBVugNS_Bi3RsZWPmbtAz3Eo1ZFAg-7o4o-Gp8IGcz%Ko8tAvSKoHelKf-uXzW60p4k1pbaJuPOuKnStp_gVy3QG7iLwsfoQf%eI_%wv9IPOGmo4KKSN2sWKfL8oCC2SqlImRzMWUMYb2TV0h9LoO-LSaD0huM7wdWaFzGffj_-32-341000_ttl=0_read__51077_71091_1643816051.22 2022-02-02 07:58:44,164 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 107 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-32-339000_vsXqDMekmQ%W4MH4FpLYbdGSWTmd41sH0nmVH7Bj7b_6km8vJcZE-idGnoz2XAergyfHSstsnJEK8ZamCs1zOB8VUyYHq6W%kY4ge4KZ2_gBJ1lER4IVmYauzbL-viP77XMKB7A3mHXSuDUGDEIBm-S7YtdLB1Jm6KwBwk0R_dlyXFjRRvXeYsBgZ6dDeMARtg6R6qlYwCKasybt6cthvbH0-32-339000_ttl=0_read__51077_71091_1643816051.22 2022-02-02 07:58:44,165 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 75 docs from 51077 to 71091 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_gYtdRKkEHiGb0N0b-32-332000_wFNE6s3VEi0e1huY-2YzWRzCbQKki4xEzJm0-4yGuyhFRDWsRW8w%fjmcjB6Ugj5Bbz9mUvQLZMol-6cu302geDEbgfn%bz%QBs0-wlTCyYmSauiG64fIrZQ3Tmy_%-32-333000_ttl=0_read__51077_71091_1643816051.23 2022-02-02 07:58:51,867 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 94.31 2022-02-02 07:59:01,898 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 94.42 2022-02-02 07:59:11,941 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 94.68 2022-02-02 07:59:21,977 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 95.15 2022-02-02 07:59:32,015 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 95.36 2022-02-02 07:59:42,056 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 95.67 2022-02-02 07:59:52,094 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.08 2022-02-02 08:00:02,125 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.19 2022-02-02 08:00:12,157 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.39 2022-02-02 08:00:22,194 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.39 2022-02-02 08:00:32,234 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.39 2022-02-02 08:00:42,270 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.55 2022-02-02 08:00:52,302 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.55 2022-02-02 08:01:02,335 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.6 2022-02-02 08:01:12,398 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.65 2022-02-02 08:01:22,438 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.86 2022-02-02 08:01:32,471 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.91 2022-02-02 08:01:42,510 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.01 2022-02-02 08:01:52,543 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.11 2022-02-02 08:02:02,576 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.16 2022-02-02 08:02:12,608 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.27 2022-02-02 08:02:22,638 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.32 2022-02-02 08:02:32,680 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.37 2022-02-02 08:02:42,710 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.42 2022-02-02 08:02:52,740 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.42 2022-02-02 08:03:02,772 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.42 2022-02-02 08:03:12,803 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.63 2022-02-02 08:03:22,835 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.63 2022-02-02 08:03:32,868 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.63 2022-02-02 08:03:42,898 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.8 2022-02-02 08:03:52,936 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.9 2022-02-02 08:04:02,967 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.06 2022-02-02 08:04:13,000 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.06 2022-02-02 08:04:23,029 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.06 2022-02-02 08:04:33,066 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.06 2022-02-02 08:04:43,101 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.12 2022-02-02 08:04:53,132 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.17 2022-02-02 08:05:03,164 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.55 2022-02-02 08:05:13,198 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.6 2022-02-02 08:05:23,230 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.87 2022-02-02 08:05:33,260 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.87 2022-02-02 08:05:41,823 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 44 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default__default_eL6jfEd%f6HD41i9_L%R0mvgcIzW8qQCP__g_fTpyppzJyAuVgAG_d3gDEpXR2xW4L3MiE0K-%mX0rOhi7favV30yWl9c-FgmXnqPwRw_ysQBCS_eRojIS3NjKiTgTEBGtKbhAVDHdBvRRd_-B8uUQFlJVCIxFYQDbs-32-317000_ttl=0_delete__26060_51077_1643816051.54 2022-02-02 08:05:43,292 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.98 2022-02-02 08:05:53,346 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.98 2022-02-02 08:06:03,382 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.09 2022-02-02 08:06:13,420 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.14 2022-02-02 08:06:14,237 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 123 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-32-327000_7H9oeaNXeAG5QH0tQbDwYiaGLgYlyYcqoDO79rSZxe3f3orsn_ST1eviXLQjWnYRz8-y4vHcqK-aaAD5YaVRBgwIQDRTLo7X6RHcRTjg0JW65q3KgTEKRNSaC3GDG%sOetjo4iztIDHRKz8Xqh-32-328000_ttl=0_delete__26060_51077_1643816051.55 2022-02-02 08:06:14,239 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 758 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-32-319000_c2Q3M0_VZE2vY6ytsJ5R94H64F%PjnctX96XyJLoqe21UXWtoZ5gR%n%dKIXdUwFccfdz6J62TXu%1dmID91Ok8UzDbd9NTvw_GV5ZFFX2F-GF5Pi--32-321000_ttl=0_delete__26060_51077_1643816051.55 2022-02-02 08:06:23,453 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.3 2022-02-02 08:06:33,493 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.35 2022-02-02 08:06:35,792 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 44 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-32-336000_dN8YZuGpPZvtVZ5mml75ZXvoH_V5F_4JoKy5PaCGpSI-Mx46sCrjDhocI%ZMLzlN9YO-ooY6WFA5TfYxwJdoEXBwFQWvqansEr8NarkgT5-32-338000_ttl=0_delete__26060_51077_1643816051.55 2022-02-02 08:06:43,528 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:06:43,979 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 43 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default__default_IJodTcMctQxMvnZP2oSR0q-p1O7lucMJJQNcDDp_NgqSOuaC8axSBC6-32-318000_ttl=0_delete__26060_51077_1643816051.55 2022-02-02 08:06:43,979 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 38 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default__default_Q1I6SGqWbuQ-RJD9HtN6a-pD3VB6T7Q76us8f2GiVxEtmy3WIIDbRKzoFM8GwzhTvuotDPfQvgkhnYFS%btjv380HuySJuc1bU_yruE7_-32-317000_ttl=0_delete__26060_51077_1643816051.55 2022-02-02 08:06:50,407 | test | ERROR | pool-6-thread-22 | [task:execute_tasks:3940] Failed to load 44 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643816050.67_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-32-327000_thlytYtgBMJ18FVpEgo2MLT722ec1SqC7Zp1cQn-6xd6tNqu7%obMX5Xw%n_Rdo5PauF-32-329000_ttl=0_delete__26060_51077_1643816051.55 2022-02-02 08:06:53,571 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:07:03,617 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:07:13,651 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:07:23,687 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:07:33,720 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:07:43,750 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:07:53,782 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:08:03,813 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:08:13,845 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.57 2022-02-02 08:08:23,881 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.73 2022-02-02 08:08:33,914 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 08:08:43,950 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 08:08:53,986 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 08:09:04,020 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 08:09:14,055 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 08:09:29,099 | test | INFO | pool-6-thread-24 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 08:09:29,118 | test | INFO | pool-6-thread-24 | [task:check:468] Rebalance completed with progress: 100% in 2122.69900012 sec 2022-02-02 08:20:19,637 | test | INFO | pool-6-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **********.............................................**....................................******...................*X | 347.400 | | 277.800 | ..............................................................................*****X | 277.800 | | 134.600 | .....................................*X | 134.600 | | 97.000 | ..........................*****X | 97.000 | | 65.300 | .................**X | 65.300 | | 53.800 | ..............*****.***..*****X | 64.800 | | 32.400 | ........*X | 32.400 | | 17.700 | ....*****X | 17.700 | | 5.800 | **.*****X | 7.600 | | 1.800 | ********....**X | 15.200 | | 11.800 | ..*****..*.**..*****X | 30.400 | | 18.700 | ....*X | 18.700 | | 15.200 | ...*****X | 15.200 | | 3.400 | **X | 3.400 | | 0.000 | **************..*X | 7.600 | | 5.800 | ********...******.....**X | 32.200 | | 25.300 | ......*****X | 25.300 | | 13.500 | ..*******X | 13.500 | | 1.700 | ********..**X | 7.600 | | 5.900 | ****X | 5.900 | | 0.000 | ********************.******.......*X | 30.400 | | 23.500 | .....*****X | 23.500 | | 0.000 | ***************************************************************************************************************************************************************.**.*****X | 7.600 | | 1.700 | **X | 1.700 | | 0.000 | **************.......**..*****X | 34.100 | | 14.900 | ...**X | 14.900 | | 5.900 | ******...........**....*****X | 57.200 | | 30.700 | .......*X | 30.700 | | 22.800 | .....*****X | 22.800 | | 5.100 | **X | 5.100 | | 0.000 | ************************************************************************.....**.*****X | 22.800 | | 5.100 | **X | 5.100 | | 0.000 | ***************************************.**.*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ***********************...**.******....**.*****X | 32.400 | | 7.300 | .**X | 7.300 | | 1.700 | *****..**X | 7.600 | | 5.900 | *****X | 5.900 | | 0.000 | ****************..**X | 7.600 | | 5.900 | ****X | 5.900 | | 1.700 | X | 1.700 | | 0.000 | **********.******...*X | 15.200 | | 11.700 | ..*****X | 11.700 | | 0.000 | ***.******...*X | 17.200 | | 13.300 | ..*****X | 13.300 | | 5.900 | **.*****X | 7.600 | | 1.700 | **X | 1.700 | | 0.000 | *****.***.*******X | 9.300 | | 5.900 | *****X | 5.900 | | 0.000 | *********************************...***.****X | 15.200 | | 3.400 | **X | 3.400 | | 0.000 | ******.**.******......*******X | 30.200 | | 5.100 | **X | 5.100 | | 0.000 | **************...**.*****X | 16.900 | | 11.000 | ..**X | 11.000 | | 5.900 | ****X | 5.900 | | 0.000 | *****************..........**...*****X | 45.600 | | 16.200 | ...**X | 16.200 | | 7.600 | .*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ******.**.*********.*****......**X | 34.400 | | 30.500 | .......*****X | 30.500 | | 23.100 | .....*X | 23.100 | | 22.600 | .....*****X | 22.600 | | 3.400 | *X | 3.400 | | 0.000 | ******.......***..**X | 32.400 | | 32.400 | ........*X | 32.400 | | 7.300 | .**X | 7.300 | | 0.000 | **********************.....**.*****X | 22.800 | | 5.100 | **X | 5.100 | | 0.000 | **************..*X | 7.600 | | 5.900 | ******.********......*...**..X | 47.600 | | 47.600 | ............*****.......**...*****X | 80.000 | | 31.300 | ........*X | 31.300 | | 17.200 | ...*****X | 17.200 | | 11.300 | ..*X | 11.300 | | 9.600 | .*****X | 9.600 | | 2.200 | **X | 2.200 | | 1.800 | ******....*******X | 15.400 | | 2.200 | *X | 2.200 | | 0.000 | *******************************...**.*****X | 17.200 | | 9.800 | .**X | 9.800 | | 9.300 | .*****...*.*******X | 21.100 | | 10.800 | ..*X | 10.800 | | 9.600 | .******.....**...*****X | 36.400 | | 30.500 | .......*X | 30.500 | | 20.800 | ....*****X | 20.800 | | 7.600 | .*******X | 7.600 | | 3.400 | X | 3.400 | | 1.700 | ********..*X | 9.600 | | 7.400 | .******...**..*****X | 25.000 | | 14.700 | ...*X | 14.700 | | 7.400 | .******.........***...****X | 49.600 | | 33.500 | ........**X | 33.500 | | 28.800 | .......*****X | 28.800 | | 12.500 | ..*X | 12.500 | | 11.600 | ..******....*X | 24.800 | | 20.800 | ....*****X | 20.800 | | 9.100 | .*******X | 9.600 | | 2.200 | **X | 2.200 | | 0.000 | ******************************.....*..**.******..**.*****X | 40.000 | | 9.100 | .*X | 9.100 | | 0.000 | ******...**...******.......*X | 45.400 | | 36.200 | .........*****X | 36.200 | | 25.700 | ......**.*****X | 30.400 | | 12.700 | ..**X | 12.700 | | 7.600 | .*****....***.*****.********X | 30.400 | | 7.100 | .*X | 7.100 | | 1.700 | ******...**..******.......**.*****X | 45.700 | | 7.400 | .**X | 7.400 | | 0.000 | **************.**.******..**.*****X | 19.200 | | 16.100 | ...**X | 16.100 | | 15.200 | ...*****X | 15.200 | | 3.500 | *X | 3.500 | | 0.000 | ******...**.*****X | 17.200 | | 4.000 | *X | 4.000 | | 0.000 | ************************************************..***************X | 9.600 | | 8.000 | .**X | 8.000 | | 7.600 | .*************X | 7.600 | | 1.800 | *X | 1.800 | | 0.000 | ******..*******X | 9.600 | | 2.200 | **X | 2.200 | | 0.000 | **************.....**...*****X | 30.400 | | 21.400 | .....*X | 21.400 | | 11.600 | ..*****X | 11.600 | | 5.800 | ***.*******X | 7.700 | | 7.600 | .******..********..**..******....*X | 41.900 | | 38.000 | .........*****X | 38.000 | | 5.800 | **X | 5.800 | | 1.700 | *****.....***.*****X | 23.100 | | 11.400 | ..*X | 11.400 | | 9.600 | .*****X | 9.600 | | 8.000 | .**..******......**X | 36.200 | | 26.400 | ......*****X | 26.400 | | 0.000 | ****************.***.******..*******X | 17.200 | | 3.900 | *X | 3.900 | | 0.000 | ******.**.*****X | 7.600 | | 1.800 | **X | 1.800 | | 0.000 | *******************************************************...**.*****X | 17.200 | | 3.900 | **X | 3.900 | | 0.000 | ********************************************************************************.***..*****.....**X | 28.500 | | 20.600 | ....*****X | 20.600 | | 13.200 | ..**.*****X | 17.200 | | 4.000 | *X | 4.000 | | 0.000 | ******.........***...*****X | 42.000 | | 9.700 | .*X | 9.700 | | 0.000 | **************...***.******..**.*****X | 24.800 | | 17.400 | ....*X | 17.400 | | 15.200 | ...*****X | 15.200 | | 9.400 | .***.******....*X | 26.600 | | 19.000 | ....*****X | 19.000 | | 0.000 | ********..***.*****...**X | 23.200 | | 19.200 | ....*****X | 19.200 | | 1.800 | *X | 1.800 | | 0.000 | ***************************************************************************.*****.......*********....**.*****X | 45.600 | | 7.600 | .*X | 7.600 | | 0.000 | ***********************....**.*****X | 19.200 | | 4.400 | **X | 4.400 | | 0.000 | ******.........**...*****X | 44.000 | | 17.600 | ....*X | 17.600 | | 9.600 | .*****X | 9.600 | | 2.200 | **X | 2.200 | | 0.000 | ******.**.*****X | 7.600 | | 1.800 | *X | 1.800 | | 0.000 | ******.**.******.**X | 11.400 | | 7.400 | .****X | 7.400 | | 0.000 | ********.......***....*******X | 40.600 | | 30.000 | .......*****X | 30.000 | | 9.400 | .*X | 9.400 | | 5.800 | *****X | 5.800 | | 0.000 | ****************....***.*****X | 19.200 | | 4.400 | *X | 4.400 | | 0.000 | ******...**.*****X | 15.200 | | 3.600 | *X | 3.600 | | 0.000 | ******.***.*****X | 7.500 | | 1.900 | *X | 1.900 | | 0.100 | ******.....**..*****X | 26.600 | | 13.400 | ..*X | 13.400 | | 5.800 | *********.*****.....***.*****X | 30.400 | | 18.800 | ....*X | 18.800 | | 13.400 | ..*****X | 13.400 | | 1.800 | *X | 1.800 | | 0.000 | *******************************..*X | 7.600 | | 5.800 | *****X | 5.800 | | 0.000 | ****X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 08:20:20,522 | test | INFO | pool-6-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ************....................******................................................................................*X | 218.300 | | 187.700 | ....................................................................................*****X | 187.700 | | 37.200 | ................*X | 37.200 | | 25.200 | ..........******...**....*****X | 41.400 | | 35.500 | ...............**X | 35.500 | | 25.300 | ..........*****X | 25.300 | | 1.700 | *X | 1.700 | | 0.000 | ***************************************..........**......*****X | 35.500 | | 29.600 | ............**X | 29.600 | | 19.400 | .......********.*****X | 21.100 | | 3.400 | *X | 3.400 | | 0.000 | ***************...**.******..*X | 15.200 | | 11.800 | ....*****X | 11.800 | | 0.000 | ************************************.....******...................**X | 53.200 | | 41.300 | .................****X | 41.300 | | 0.000 | ****************************************************************************************************************************************************************..**.*****X | 7.600 | | 1.700 | **X | 1.700 | | 0.000 | *******************************......................**......*X | 62.800 | | 62.800 | ...........................*X | 62.800 | | 62.800 | ...........................*X | 62.800 | | 31.700 | .............*X | 31.700 | | 22.800 | .........*****X | 22.800 | | 11.100 | ....**X | 11.100 | | 7.600 | ..*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ***************************************...*X | 7.600 | | 5.900 | .*********.*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ******..........**...*****X | 30.400 | | 6.800 | ..**X | 6.800 | | 0.000 | ****************.******.....**X | 15.200 | | 11.800 | ....*****X | 11.800 | | 7.400 | ..**.******.......**..*****X | 30.400 | | 6.800 | ..*X | 6.800 | | 0.000 | ***********************.....**.******.......**.**X | 32.400 | | 32.400 | .............*X | 32.400 | | 32.400 | .............X | 32.400 | | 7.300 | ..*X | 7.300 | | 0.000 | **************************************************************************************************..**...******...........*X | 36.300 | | 32.900 | ..............*****X | 32.900 | | 15.200 | .....*******X | 15.200 | | 3.400 | **X | 3.400 | | 0.000 | **********************........***..****X | 22.800 | | 5.100 | .**X | 5.100 | | 0.000 | **************............**..*****X | 32.400 | | 13.200 | .....**X | 13.200 | | 5.900 | .*****X | 5.900 | | 0.000 | ********........**..*****X | 22.800 | | 5.100 | .**X | 5.100 | | 0.000 | **************..**.******.....**..*****X | 22.800 | | 5.100 | .**X | 5.100 | | 0.000 | *************************....*****.................**X | 45.900 | | 38.600 | ................**X | 38.600 | | 38.600 | ................**X | 38.600 | | 19.400 | .......**...*****X | 24.600 | | 18.600 | .......*X | 18.600 | | 15.100 | .....******...*******X | 21.100 | | 3.500 | **X | 3.500 | | 0.000 | ******..**.******.........**..*X | 32.400 | | 32.400 | .............***X | 32.400 | | 13.200 | .....**X | 13.200 | | 7.600 | ..****X | 7.600 | | 1.700 | **X | 1.700 | | 0.000 | ******........**.....*****X | 29.900 | | 28.400 | ............*X | 28.400 | | 17.700 | .......*****X | 17.700 | | 5.800 | .***.****X | 7.600 | | 3.400 | X | 3.400 | | 1.800 | *X | 1.800 | | 0.000 | ******...................**......*****X | 55.200 | | 18.300 | .......*X | 18.300 | | 7.600 | ..********..******.....**X | 22.800 | | 19.400 | .......*****.***..*****X | 24.800 | | 13.000 | ....*X | 13.000 | | 9.600 | ...*****X | 9.600 | | 2.200 | **X | 2.200 | | 0.000 | ********.******.........**..*****X | 26.800 | | 25.200 | ..........*X | 25.200 | | 22.600 | .........*****X | 22.600 | | 3.500 | **X | 3.500 | | 0.000 | **********************...........**.....******...*X | 42.100 | | 36.100 | ...............*****X | 36.100 | | 5.200 | .**X | 5.200 | | 0.000 | ********.******........*X | 21.100 | | 19.400 | .......******......**...*****X | 38.000 | | 14.400 | .....*X | 14.400 | | 9.400 | ...******.........***.****X | 30.600 | | 24.600 | ..........X | 24.600 | | 5.700 | .*X | 5.700 | | 0.000 | ********.....******...............................*******X | 80.700 | | 51.300 | ......................**X | 51.300 | | 44.000 | ...................*****X | 44.000 | | 11.500 | ....*X | 11.500 | | 7.600 | ..*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | ******...**..******...**X | 17.700 | | 15.000 | .....****X | 15.000 | | 13.800 | .....X | 13.800 | | 13.500 | .....********.*******X | 17.200 | | 4.000 | *X | 4.000 | | 0.000 | ********..******...............*X | 38.100 | | 36.400 | ...............******.**...*****X | 44.000 | | 42.400 | ..................**X | 42.400 | | 42.000 | ..................****X | 42.000 | | 37.700 | ................***.*****..**X | 44.600 | | 38.100 | ................*****X | 38.100 | | 5.700 | .*X | 5.700 | | 0.000 | **************.........**...*****X | 26.800 | | 6.100 | .**X | 6.100 | | 0.000 | ******************************.....................***.......*****X | 63.000 | | 21.700 | ........*X | 21.700 | | 5.800 | .*****X | 5.800 | | 0.000 | *********************************........**...*****X | 24.800 | | 5.600 | .**X | 5.600 | | 0.000 | *****..***.******.*******X | 9.600 | | 2.200 | **.******............**X | 30.700 | | 26.700 | ...........*****X | 26.700 | | 9.100 | ...*******X | 9.600 | | 2.200 | *X | 2.200 | | 0.000 | ******............**.....*****X | 38.300 | | 30.800 | .............**X | 30.800 | | 20.900 | ........*****X | 20.900 | | 1.800 | *X | 1.800 | | 0.000 | ******......**.*****X | 17.200 | | 17.100 | ......*******X | 17.200 | | 4.000 | **X | 4.000 | | 0.000 | ******........**...******.......**......******..*X | 57.900 | | 40.000 | .................*****X | 40.000 | | 1.800 | *X | 1.800 | | 0.000 | **********************************....*****........................***...*****X | 68.800 | | 49.800 | .....................*X | 49.800 | | 33.800 | ..............*****X | 33.800 | | 17.400 | ......**...*****X | 22.800 | | 5.400 | .**X | 5.400 | | 0.000 | ******............**....*****X | 36.200 | | 15.600 | ......*X | 15.600 | | 5.800 | .*****X | 5.800 | | 0.000 | ************************************************************.******..............**.*****X | 36.000 | | 11.100 | ....**X | 11.100 | | 7.600 | ..*****X | 7.600 | | 1.800 | *X | 1.800 | | 0.000 | ******...**.*************X | 9.600 | | 2.200 | **X | 2.200 | | 0.000 | ***************************************.....**.*****X | 15.200 | | 3.600 | **.******..........**X | 26.800 | | 20.600 | ........********...*****X | 26.800 | | 12.000 | ....*X | 12.000 | | 9.300 | ...******..........................***......*****X | 80.300 | | 17.300 | ......*X | 17.300 | | 0.000 | ******************************************.*****......**X | 17.200 | | 15.000 | .....*****X | 15.000 | | 7.600 | ..********.*X | 9.600 | | 9.200 | ...******..***.*****X | 15.400 | | 2.200 | ********......*X | 17.200 | | 13.300 | .....*****X | 13.300 | | 0.000 | ********..***..******..*X | 15.200 | | 13.500 | .....*****X | 13.500 | | 1.800 | ********............***..*****X | 30.600 | | 5.800 | .*X | 5.800 | | 1.800 | ******...*X | 7.600 | | 5.800 | .******.***....*****..............********X | 47.800 | | 22.900 | .........*X | 22.900 | | 13.300 | .....*****X | 13.300 | | 0.000 | **************************************************..**.*****X | 7.600 | | 1.800 | **X | 1.800 | | 0.000 | **************......**..*****X | 19.000 | | 11.600 | ....*X | 11.600 | | 9.800 | ...******.......**X | 24.600 | | 22.800 | .........****X | 22.800 | | 2.200 | **X | 2.200 | | 0.000 | *****.................***......*****..**X | 55.600 | | 53.400 | .......................*****X | 53.400 | | 17.000 | ......*X | 17.000 | | 7.600 | ..*****X | 7.600 | | 1.800 | *X | 1.800 | | 0.000 | **************.............***..*****X | 34.600 | | 6.600 | ..*X | 6.600 | | 0.000 | **************..**.******.....***.....*****..**X | 34.000 | | 25.000 | ..........********..*****X | 30.400 | | 7.200 | ..*X | 7.200 | | 0.000 | **************..***.******.*******X | 9.600 | | 8.000 | ..*X | 8.000 | | 7.600 | ..*************************.******..*X | 15.200 | | 13.400 | .....*****X | 13.400 | | 1.800 | **X | 1.800 | | 0.000 | **X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 08:20:21,390 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................***..................*****..............................................................*X | 48540.700 | | 48422.900 | ..................................................................................................***X | 48422.900 | | 48422.900 | ..................................................................................................*X | 48422.900 | | 21558.700 | ...........................................**X | 21558.700 | | 16180.900 | ................................****X | 16180.900 | | 11986.400 | .......................***.******................*X | 20021.300 | | 19997.800 | ........................................*****X | 19997.800 | | 13818.100 | ...........................*X | 13818.100 | | 12757.900 | .........................*****X | 12757.900 | | 7262.200 | .............*X | 7262.200 | | 5886.300 | ...........*X | 5886.300 | | 5886.300 | ...........**X | 5886.300 | | 5886.300 | ...........*....*X | 8127.400 | | 8127.400 | ...............*.******.**.X | 9320.100 | | 9320.100 | ..................****X | 9320.100 | | 6342.600 | ............*X | 6342.600 | | 5210.700 | .........*****X | 5210.700 | | 4210.300 | .......***.*****...**X | 5945.100 | | 5510.300 | ..........*X | 5510.300 | | 5510.300 | ..........***X | 5510.300 | | 3209.400 | .....**.******....*******X | 5582.800 | | 5214.600 | .........**X | 5214.600 | | 5163.300 | .........****X | 5163.300 | | 5133.100 | .........X | 5133.100 | | 4051.100 | .......*X | 4051.100 | | 3905.300 | .......******.*X | 4665.600 | | 4616.500 | ........********.******........***..****X | 9997.900 | | 6939.400 | .............**X | 6939.400 | | 5692.400 | ..........*****X | 5692.400 | | 3984.100 | .......*X | 3984.100 | | 3606.900 | ......*****X | 3606.900 | | 1078.500 | .*X | 1078.500 | | 487.700 | *****X | 487.700 | | 399.100 | **X | 399.100 | | 383.300 | ****X | 383.300 | | 368.500 | **X | 368.500 | | 340.300 | *****X | 340.300 | | 321.500 | *******X | 357.600 | | 322.900 | *X | 322.900 | | 276.100 | *****X | 276.100 | | 186.400 | **********X | 218.900 | | 192.900 | *****X | 192.900 | | 188.400 | **********X | 280.000 | | 267.200 | *************X | 356.900 | | 338.700 | *******X | 349.000 | | 315.800 | *X | 315.800 | | 295.900 | ********X | 300.500 | | 280.900 | *****X | 280.900 | | 168.100 | ***************X | 406.400 | | 275.900 | *X | 275.900 | | 252.000 | **************X | 391.800 | | 383.400 | *X | 383.400 | | 370.900 | *****X | 370.900 | | 268.000 | *X | 268.000 | | 266.300 | **************X | 432.900 | | 381.500 | *X | 381.500 | | 351.000 | ******.****X | 889.100 | | 889.100 | ***.......*X | 4046.400 | | 4046.400 | .......*..X | 5333.700 | | 5333.700 | .........***X | 5333.700 | | 5143.100 | .........**X | 5143.100 | | 4356.100 | .......*****X | 4356.100 | | 2602.700 | ....*X | 2602.700 | | 2299.400 | ...*****X | 2299.400 | | 1513.400 | ..*X | 1513.400 | | 1469.800 | ..******....***.****X | 3979.300 | | 3211.900 | .....X | 3211.900 | | 3211.900 | .....*X | 3211.900 | | 3191.700 | .....***X | 3191.700 | | 3191.700 | .....**.********.*X | 4152.200 | | 3953.300 | .......**X | 3953.300 | | 3953.300 | .......**X | 3953.300 | | 1753.600 | ..**X | 1753.600 | | 1241.000 | .*****X | 1241.000 | | 1228.100 | .********..**.*****X | 2520.300 | | 1890.200 | ..*X | 1890.200 | | 1791.100 | ..*****X | 1791.100 | | 1327.000 | .**X | 1327.000 | | 1151.100 | .*************X | 1298.900 | | 899.300 | *X | 899.300 | | 834.400 | *********.****X | 987.100 | | 686.300 | **X | 686.300 | | 585.200 | **************.*******X | 1093.600 | | 909.800 | ***.*****..**X | 2204.100 | | 2061.500 | ...*****X | 2061.500 | | 1305.200 | .**..***X | 2071.100 | | 2071.100 | ...**.............**..X | 9556.600 | | 9556.600 | ..................*X | 9556.600 | | 9556.600 | ..................*X | 9556.600 | | 9556.600 | ..................X | 9556.600 | | 5684.000 | ..........*X | 5684.000 | | 4107.300 | .......****X | 4107.300 | | 4107.300 | .......X | 4107.300 | | 2640.100 | ....**.******..**X | 4281.400 | | 3976.400 | .......**X | 3976.400 | | 3976.400 | .......*X | 3976.400 | | 2402.500 | ...**X | 2402.500 | | 2081.000 | ...******.**.*****X | 3103.700 | | 2423.800 | ...*X | 2423.800 | | 2289.600 | ...*****X | 2289.600 | | 1589.100 | ..**X | 1589.100 | | 1146.700 | .****X | 1146.700 | | 940.800 | X | 940.800 | | 770.700 | ********.********..*X | 2232.700 | | 2182.000 | ...*****X | 2182.000 | | 1089.300 | .**X | 1089.300 | | 931.700 | ******.**..******..*X | 3383.300 | | 2698.400 | ....*****X | 2698.400 | | 1478.800 | ..********...*********.....**..******..*X | 7752.700 | | 7268.300 | .............*****X | 7268.300 | | 3127.300 | .....**X | 3127.300 | | 2586.200 | ....****X | 2586.200 | | 2564.900 | ....**X | 2564.900 | | 2353.600 | ...******....**..******.*X | 5375.700 | | 4600.700 | ........*****X | 4600.700 | | 3094.900 | .....**X | 3094.900 | | 3086.200 | .....****X | 3086.200 | | 2513.800 | ....*********....*X | 4843.800 | | 4726.300 | ........*****X | 4726.300 | | 3028.100 | .....*X | 3028.100 | | 2819.100 | ....******....***..*****..*********.....**.*****X | 9399.300 | | 8877.900 | .................*X | 8877.900 | | 8570.200 | ................*****X | 8570.200 | | 6743.700 | ............**X | 6743.700 | | 6395.200 | ............*****X | 6395.200 | | 4430.400 | ........*X | 4430.400 | | 3235.600 | .....******.....**....******.**X | 8137.800 | | 7750.300 | ..............*****...***.*****X | 9319.900 | | 7333.100 | ..............*X | 7333.100 | | 6472.100 | ............*****X | 6472.100 | | 4098.600 | .......*X | 4098.600 | | 3617.100 | ......******.......***..*****......********X | 11070.200 | | 8633.000 | ................*X | 8633.000 | | 7796.400 | ...............******.......**....******....*X | 15058.900 | | 13607.600 | ...........................*****X | 13607.600 | | 9331.200 | ..................**X | 9331.200 | | 9179.000 | .................****X | 9179.000 | | 8133.300 | ...............X | 8133.300 | | 5081.100 | .........*X | 5081.100 | | 4365.100 | .......******..........**..*****X | 10132.800 | | 9973.400 | ...................*X | 9973.400 | | 9726.800 | ...................******...**.*****X | 11770.600 | | 6648.900 | ............**X | 6648.900 | | 5606.400 | ..........********.X | 6050.800 | | 6050.800 | ...........*****...*****X | 7728.300 | | 7728.300 | ..............**.................**...*****X | 17325.400 | | 15073.800 | ..............................X | 15073.800 | | 14345.300 | ............................*X | 14345.300 | | 13352.400 | ..........................****X | 13352.400 | | 10720.300 | .....................*********.*X | 11637.400 | | 10954.500 | .....................*****X | 10954.500 | | 7655.400 | ..............*X | 7655.400 | | 7418.300 | ..............******..........***...****X | 13887.200 | | 11982.800 | .......................**X | 11982.800 | | 10989.300 | .....................*****X | 10989.300 | | 5062.800 | .........*X | 5062.800 | | 4009.900 | .......******..**.******.....X | 7849.600 | | 7849.600 | ...............**..X | 8937.200 | | 8937.200 | .................**X | 8937.200 | | 8937.200 | .................*...**X | 10589.800 | | 10330.400 | ....................*****X | 10330.400 | | 6509.400 | ............*X | 6509.400 | | 6078.300 | ...........******.....**.****X | 9049.500 | | 9049.500 | .................X | 9049.500 | | 8468.900 | ................********........***...******....**.*****X | 16239.600 | | 15743.800 | ...............................*X | 15743.800 | | 14513.700 | ............................*****X | 14513.700 | | 9436.800 | ..................*X | 9436.800 | | 9401.500 | ..................******....*.**.*****X | 12165.900 | | 11915.800 | .......................*X | 11915.800 | | 11801.800 | .......................*****X | 11801.800 | | 9678.300 | ..................**.******.............**...*************X | 17942.400 | | 16971.200 | .................................**X | 16971.200 | | 16190.900 | ................................*****X | 16190.900 | | 8450.900 | ................*X | 8450.900 | | 6898.300 | .............******.**.X | 7963.600 | | 7963.600 | ...............**X | 7963.600 | | 7963.600 | ...............**.........**.*****X | 13069.700 | | 6481.700 | ............**X | 6481.700 | | 4964.000 | .........*******X | 5133.100 | | 4935.400 | .........******..........**...*****X | 11364.700 | | 6464.900 | ............*X | 6464.900 | | 5046.800 | .........***X | 5046.800 | | 5046.800 | .........**.....*..**..*****..................***.......****X | 21366.100 | | 20330.700 | ........................................**X | 20330.700 | | 18388.700 | ....................................*****X | 18388.700 | | 10744.000 | .....................*X | 10744.000 | | 10108.000 | ...................******............**...******..................*******X | 26146.500 | | 18037.000 | ....................................*X | 18037.000 | | 17104.900 | ..................................******......***....*******X | 22160.800 | | 20374.800 | ........................................****X | 20374.800 | | 10922.100 | .....................**X | 10922.100 | | 9344.900 | ..................X | 9344.900 | | 9344.900 | ..................*****........**...******.............**..*****X | 22053.000 | | 7677.600 | ..............*X | 7677.600 | | 4083.300 | .......******....*X | 6003.800 | | 6003.800 | ...........*.*****X | 6573.000 | | 1986.200 | ...*X | 1986.200 | | 895.200 | ******.**..******...........X | 7632.200 | | 7632.200 | ..............*.*****X | 7998.800 | | 6092.700 | ...........*X | 6092.700 | | 6092.700 | ...........X | 6092.700 | | 5869.600 | ...........******.......**...******..................**..*****X | 20636.900 | | 10168.900 | ...................*X | 10168.900 | | 8115.400 | ...............*X | 8115.400 | | 8115.400 | ...............***X | 8115.400 | | 5352.600 | ..........**X | 5352.600 | | 4841.800 | ........*****X | 4841.800 | | 3621.700 | ......*X | 3621.700 | | 3456.000 | ......*****X | 3456.000 | | 1541.700 | ..*X | 1541.700 | | 1424.000 | .*****X | 1424.000 | | 1413.800 | .**X | 1413.800 | | 1409.900 | .*****............***....*****X | 8864.300 | | 5993.300 | ...........*X | 5993.300 | | 5469.900 | ..........******.............**.*****X | 12305.000 | | 11299.200 | ......................***.****X | 11824.900 | | 9193.500 | .................**X | 9193.500 | | 7224.800 | .............******......**..*****X | 11130.200 | | 9699.600 | ..................*X | 9699.600 | | 8976.800 | .................*****X | 8976.800 | | 2060.800 | ...**X | 2060.800 | | 705.800 | *****.............***....******....*X | 10777.000 | | 10291.500 | ....................*****X | 10291.500 | | 8668.200 | ................X | 8668.200 | | 8668.200 | ................*.******..........***.****X | 14338.300 | | 11242.300 | ......................**X | 11242.300 | | 11127.300 | .....................******........**.******.......................**....*****X | 28274.400 | | 13129.700 | ..........................*X | 13129.700 | | 10344.400 | ....................******.*X | 10981.400 | | 10743.700 | .....................*****X | 10743.700 | | 10671.600 | ....................**X | 10671.600 | | 10424.800 | ....................******.**..******..*X | 12726.600 | | 11503.300 | ......................*****X | 11503.300 | | 6115.000 | ...........*X | 6115.000 | | 4738.600 | ........******.........***.....******......*X | 14227.200 | | 13286.100 | ..........................*****X | 13286.100 | | 4859.600 | .........*X | 4859.600 | | 4486.000 | ........******..........*X | 9624.500 | | 9589.900 | ..................*****X | 9589.900 | | 9481.800 | ..................*******X | 9593.300 | | 6982.200 | .............**X | 6982.200 | | 6464.300 | ............*****X | 6464.300 | | 3690.600 | ......*X | 3690.600 | | 3098.800 | .....*****X | 3098.800 | | 2879.300 | ....**.********X | 2987.300 | | 2869.900 | ....****X | 2869.900 | | 2558.300 | ....**X | 2558.300 | | 2485.600 | ....*****X | 2485.600 | | 2073.300 | ...********.....**.******.***.****X | 6108.600 | | 4330.700 | .......**X | 4330.700 | | 3832.700 | ......******....**.******.......**..*****X | 10355.700 | | 6625.100 | ............*X | 6625.100 | | 5726.900 | ..........*****X | 5726.900 | | 2213.900 | ...**X | 2213.900 | | 1590.300 | ..****X | 1590.300 | | 1506.800 | ..*********.*******X | 2397.300 | | 2154.000 | ...*X | 2154.000 | | 1815.700 | ..******.***.*****.....***...******...........**..******.......*X | 16358.700 | | 15615.500 | ...............................*****X | 15615.500 | | 12155.600 | ........................********....**X | 14541.300 | | 14376.000 | ............................****X | 14376.000 | | 10833.300 | .....................**X | 10833.300 | | 10152.900 | ...................*****X | 10152.900 | | 6090.300 | ...........*X | 6090.300 | | 5370.600 | ..........******.....*******X | 8187.200 | | 7950.400 | ...............********.***.*******X | 8835.600 | | 8654.300 | ................*****X | 8654.300 | | 8179.400 | ...............*X | 8179.400 | | 8074.100 | ...............*****X | 8074.100 | | 4672.200 | ........*X | 4672.200 | | 3676.200 | ......******......***..*****.........***.*****X | 12591.000 | | 10670.600 | ....................*X | 10670.600 | | 9805.600 | ...................*****X | 9805.600 | | 4384.200 | ........*X | 4384.200 | | 3878.100 | ......******...........***..****X | 9776.900 | | 5449.400 | ..........**X | 5449.400 | | 3921.900 | .......*****X | 3921.900 | | 2912.100 | ....**..******.........**.*****X | 8593.200 | | 8190.800 | ...............***..******.....*******X | 11390.000 | | 10139.800 | ...................*X | 10139.800 | | 10011.400 | ...................*****X | 10011.400 | | 7504.100 | ..............*X | 7504.100 | | 6816.700 | .............******....***.******..********.........**.*****X | 15080.100 | | 6722.200 | ............*X | 6722.200 | | 5604.000 | ..........******.........**..******......*******X | 13791.800 | | 13044.800 | .........................**X | 13044.800 | | 12851.600 | .........................*****X | 12851.600 | | 8289.900 | ................*X | 8289.900 | | 7361.300 | ..............*****X | 7361.300 | | 4735.000 | ........*X | 4735.000 | | 4163.000 | .......******.********X | 4825.700 | | 4802.800 | ........**.******...........**..*****X | 11432.300 | | 10567.100 | ....................**.******.........**.*****X | 15930.100 | | 8256.300 | ................*X | 8256.300 | | 6447.300 | ............******..................***.....*************...............***..****X | 25992.100 | | 13114.600 | ..........................**X | 13114.600 | | 9215.400 | .................*****X | 9215.400 | | 5445.200 | ..........*X | 5445.200 | | 5156.300 | .........******.....**...*****X | 8771.300 | | 8114.500 | ...............**X | 8114.500 | | 7052.500 | .............*****X | 7052.500 | | 3283.500 | .....**.******................**...******...**.*****X | 14624.900 | | 11721.500 | .......................*X | 11721.500 | | 11400.100 | ......................******......***..******.*X | 15784.700 | | 15170.600 | ..............................*****X | 15170.600 | | 10292.300 | ....................*X | 10292.300 | | 9846.000 | ...................******....**.******....***.****X | 14565.500 | | 11963.700 | .......................**X | 11963.700 | | 10705.100 | .....................*****X | 10705.100 | | 5882.300 | ...........*X | 5882.300 | | 5431.000 | ..........**************...***.*****....***.******..........**.*****X | 15339.300 | | 14724.000 | .............................*X | 14724.000 | | 14697.400 | .............................***X | 14697.400 | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 08:20:22,690 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 14697.400 | ...................................................................................................X | 14697.400 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 08:20:47,117 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 08:21:19,920 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 08:21:34,421 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 08:21:34,421 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 08:21:45,707 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 08:21:45,707 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 08:21:57,009 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 08:21:57,010 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 08:22:15,801 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 46000 | 9.77 GiB | 237.76 MiB | 334.52 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 4.88 GiB | 466.40 MiB | 459.46 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 30021250 | 2.50 GiB | 1.49 GiB | 59.41 GiB | 38.0635050173 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 08:22:16,289 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 08:22:16,289 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 08:22:22,782 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 08:22:22,782 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 08:22:29,240 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 45% 2022-02-02 08:22:29,240 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 2% 2022-02-02 08:23:22,778 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 08:23:23,214 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 08:23:23,214 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 08:23:53,354 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 08:23:53,793 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 08:23:53,793 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 08:24:23,904 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 08:24:24,331 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 08:24:24,331 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 08:27:25,700 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'e63cd9e29b3aa35c1ca9117a93b7cc0a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d308561aa0938efd4cc7e8252770e46c', u'status': u'notRunning'} - rebalance failed 2022-02-02 08:27:25,726 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643819244531L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:27:24.531Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = a7a1467610939d6d8390a1879826ee33'} 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643819244530L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:27:24.530Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643819239459L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:27:19.459Z', u'text': u'Shutting down bucket "bucket2" on \'ns_1@172.23.105.36\' for deletion'} 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643819075574L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:24:35.574Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643819064527L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:24:24.527Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = a7a1467610939d6d8390a1879826ee33"} 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643819064520L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T08:24:24.520Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 08:27:25,726 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643819059760L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:24:19.760Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 08:27:25,727 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.177', u'tstamp': 1643819024060L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:23:44.060Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 08:27:25,727 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643818999045L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:23:19.045Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 08:27:25,727 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643818996947L, u'shortText': u'message', u'serverTime': u'2022-02-02T08:23:16.947Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance 2022-02-02 08:27:25,729 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'e63cd9e29b3aa35c1ca9117a93b7cc0a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d308561aa0938efd4cc7e8252770e46c', u'status': u'notRunning'} - rebalance failed progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'e63cd9e29b3aa35c1ca9117a93b7cc0a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d308561aa0938efd4cc7e8252770e46c', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 8984.540s During the test, Remote Connections: 259, Disconnections: 259 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_3 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_delta_recovery_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_4', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_delta_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 4, 'randomize_value': 'True'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 08:27:31,635 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 08:27:32,006 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 08:29:40,953 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.705000162 seconds to complete 2022-02-02 08:29:40,960 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 08:29:46,040 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x2c5515 type=client open_count=1 channel=[id: 0xf8b10ea1, 0.0.0.0/0.0.0.0:50342] timeout=300.0>) 2022-02-02 08:29:49,858 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x2c5516 type=client open_count=1 channel=[id: 0x95c160b5, 0.0.0.0/0.0.0.0:59067] timeout=300.0>) 2022-02-02 08:29:51,868 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x2c5517 type=client open_count=1 channel=[id: 0x349d4fba, 0.0.0.0/0.0.0.0:41760] timeout=300.0>) 2022-02-02 08:29:53,878 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x2c5518 type=client open_count=1 channel=[id: 0x7ed77681, 0.0.0.0/0.0.0.0:41452] timeout=300.0>) 2022-02-02 08:29:55,888 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x2c5519 type=client open_count=1 channel=[id: 0xa014c227, 0.0.0.0/0.0.0.0:41377] timeout=300.0>) 2022-02-02 08:30:01,618 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x2c551a type=client open_count=1 channel=[id: 0xe6e2d6ff, 0.0.0.0/0.0.0.0:55923] timeout=300.0>) 2022-02-02 08:30:03,628 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x2c551b type=client open_count=1 channel=[id: 0xad6dac2b, 0.0.0.0/0.0.0.0:47755] timeout=300.0>) 2022-02-02 08:30:05,638 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x2c551c type=client open_count=1 channel=[id: 0x934ef747, 0.0.0.0/0.0.0.0:56903] timeout=300.0>) 2022-02-02 08:30:07,650 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x2c551d type=client open_count=1 channel=[id: 0xc6dd0947, 0.0.0.0/0.0.0.0:39990] timeout=300.0>) 2022-02-02 08:30:12,904 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x2c551e type=client open_count=1 channel=[id: 0x2619a051, 0.0.0.0/0.0.0.0:58347] timeout=300.0>) 2022-02-02 08:30:14,914 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x2c551f type=client open_count=1 channel=[id: 0x50d89435, 0.0.0.0/0.0.0.0:45686] timeout=300.0>) 2022-02-02 08:30:16,924 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x2c5520 type=client open_count=1 channel=[id: 0x2d34e513, 0.0.0.0/0.0.0.0:48685] timeout=300.0>) 2022-02-02 08:30:18,934 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x2c5521 type=client open_count=1 channel=[id: 0x6c0abcce, 0.0.0.0/0.0.0.0:36463] timeout=300.0>) 2022-02-02 08:30:24,505 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x2c5522 type=client open_count=1 channel=[id: 0x3b7c9b2d, 0.0.0.0/0.0.0.0:39247] timeout=300.0>) 2022-02-02 08:30:26,515 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x2c5523 type=client open_count=1 channel=[id: 0x94ffd25d, 0.0.0.0/0.0.0.0:38844] timeout=300.0>) 2022-02-02 08:30:28,525 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x2c5524 type=client open_count=1 channel=[id: 0x13035fad, 0.0.0.0/0.0.0.0:46248] timeout=300.0>) 2022-02-02 08:30:30,536 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x2c5525 type=client open_count=1 channel=[id: 0xb5681f52, 0.0.0.0/0.0.0.0:34646] timeout=300.0>) 2022-02-02 08:30:32,546 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x2c5526 type=client open_count=1 channel=[id: 0x24981b46, 0.0.0.0/0.0.0.0:36701] timeout=300.0>) 2022-02-02 08:30:37,618 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x2c5527 type=client open_count=1 channel=[id: 0x4b87bc05, 0.0.0.0/0.0.0.0:39672] timeout=300.0>) 2022-02-02 08:30:39,628 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x2c5528 type=client open_count=1 channel=[id: 0xda89be34, 0.0.0.0/0.0.0.0:60597] timeout=300.0>) 2022-02-02 08:30:41,638 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x2c5529 type=client open_count=1 channel=[id: 0x2735a4ae, 0.0.0.0/0.0.0.0:33446] timeout=300.0>) 2022-02-02 08:30:43,648 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c552a type=client open_count=1 channel=[id: 0x6343c784, 0.0.0.0/0.0.0.0:48847] timeout=300.0>) 2022-02-02 08:30:49,423 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c552b type=client open_count=1 channel=[id: 0xb9f73122, 0.0.0.0/0.0.0.0:37233] timeout=300.0>) 2022-02-02 08:30:51,433 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c552c type=client open_count=1 channel=[id: 0x06a54a67, 0.0.0.0/0.0.0.0:60622] timeout=300.0>) 2022-02-02 08:30:53,443 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c552d type=client open_count=1 channel=[id: 0x0b0206f7, 0.0.0.0/0.0.0.0:59450] timeout=300.0>) 2022-02-02 08:30:55,453 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c552e type=client open_count=1 channel=[id: 0x299f1d15, 0.0.0.0/0.0.0.0:37371] timeout=300.0>) 2022-02-02 08:30:57,463 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c552f type=client open_count=1 channel=[id: 0x335ae285, 0.0.0.0/0.0.0.0:58038] timeout=300.0>) 2022-02-02 08:30:59,473 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x2c5530 type=client open_count=1 channel=[id: 0x5a759619, 0.0.0.0/0.0.0.0:59304] timeout=300.0>) 2022-02-02 08:31:01,483 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x2c5531 type=client open_count=1 channel=[id: 0x630f3c89, 0.0.0.0/0.0.0.0:36915] timeout=300.0>) 2022-02-02 08:31:06,875 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x2c5532 type=client open_count=1 channel=[id: 0x7dcc57d5, 0.0.0.0/0.0.0.0:56869] timeout=300.0>) 2022-02-02 08:31:08,884 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x2c5533 type=client open_count=1 channel=[id: 0xd79c4a42, 0.0.0.0/0.0.0.0:54909] timeout=300.0>) 2022-02-02 08:31:10,894 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x2c5534 type=client open_count=1 channel=[id: 0xb3860e1a, 0.0.0.0/0.0.0.0:49852] timeout=300.0>) 2022-02-02 08:31:12,904 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 08:31:27,778 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 08:31:40,335 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 08:31:40,335 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 08:31:40,351 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 08:31:53,125 | test | INFO | pool-7-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.67835671343 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 08:31:53,144 | test | INFO | pool-7-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 08:32:08,178 | test | INFO | pool-7-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 08:32:08,191 | test | INFO | pool-7-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0650000572 sec 2022-02-02 08:32:08,312 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.675929402929 | 11.45 GiB | 10.67 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.626409421198 | 11.45 GiB | 10.65 GiB | 24.25 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.5795411809 | 11.45 GiB | 10.66 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.213005888986 | 11.45 GiB | 10.75 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 2.17990478577 | 11.45 GiB | 10.58 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 08:32:08,312 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 08:32:08,312 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 08:32:08,480 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 08:32:08,940 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 08:32:09,270 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 89.89 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 08:32:09,270 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 08:32:18,733 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 08:32:18,740 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 09:25:32,117 | test | INFO | pool-7-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 846.300 | ...............................................................................**....................******X | 1050.000 | | 203.700 | ..................*X | 203.700 | | 0.000 | *************************************************************************************************************************************************************************************************************............**....******.......*X | 250.000 | | 240.300 | .....................*****X | 240.300 | | 159.700 | ..............**X | 159.700 | | 140.300 | ............****X | 140.300 | | 59.700 | ....**X | 59.700 | | 40.300 | ..*****X | 40.300 | | 0.000 | *******************************************************************************************************************************************************************************************************************************************************...........**...******...................**.....*****X | 400.000 | | 198.500 | .................**X | 198.500 | | 150.000 | .............****X | 150.000 | | 29.100 | .**X | 29.100 | | 0.000 | *************************************************************************************************************************************************************************************************************************************************************...........**....*****X | 159.700 | | 119.400 | ..........*X | 119.400 | | 90.300 | .......*********..****X | 109.700 | | 69.400 | .....**X | 69.400 | | 50.000 | ...*******X | 50.000 | | 40.300 | ..*****X | 40.300 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************....*X | 50.000 | | 40.300 | ..********.....******...........**X | 209.700 | | 161.200 | ..............****X | 161.200 | | 0.000 | *********...**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *******************************...**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************************************.......***....*****....**X | 159.700 | | 130.600 | ...........*****X | 130.600 | | 9.700 | *X | 9.700 | | 0.000 | *****************...*****................**X | 200.000 | | 161.200 | ..............*****X | 161.200 | | 0.000 | *********************************************************************************************************************************************************************************************************************************************************************************...........***...*****X | 150.000 | | 29.100 | .*X | 29.100 | | 0.000 | **************...***.****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ********************************************************************************************************************************************************************************************************************.**.******.**X | 38.200 | | 28.500 | .*****.***..******....*X | 100.000 | | 90.300 | .......*****X | 90.300 | | 9.700 | ********............***.****X | 140.300 | | 19.400 | **X | 19.400 | | 0.000 | ***********************************************************************************************************************************************************************************...**.*****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ******...**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ******.......***..*************X | 100.000 | | 19.400 | ********........**********X | 100.000 | | 80.600 | ......****X | 80.600 | | 40.300 | ..***.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ***************************************************************************************************************************************************************************************************...*....**...******....*X | 150.000 | | 140.300 | ............*****X | 140.300 | | 19.400 | *X | 19.400 | | 0.000 | ********************************************************************************************************************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 09:25:32,148 | test | INFO | pool-7-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.........................................**..........*****X | 200.000 | | 38.800 | .........**X | 38.800 | | 0.000 | ********************************************************************************************************************************************************************************************************************....................**.....*****X | 100.000 | | 59.700 | ..............**********X | 59.700 | | 40.300 | .........*****X | 40.300 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************..........**..**************.....................**.....******...........***..****X | 200.000 | | 79.100 | ...................**X | 79.100 | | 50.000 | ...........****X | 50.000 | | 9.700 | .**X | 9.700 | | 0.000 | *****************************************************************************************************************************************************************************************************************************************************..........**..*****X | 50.000 | | 9.700 | .*X | 9.700 | | 0.000 | ********..******....................................................***.......****X | 240.300 | | 188.800 | ...............................................X | 188.800 | | 38.800 | .........*X | 38.800 | | 0.000 | *************************************************************************************************************************************************************************************************************************************************************..........**...........******..........................................**X | 246.200 | | 215.000 | ......................................................*****X | 215.000 | | 134.400 | .................................**....*****X | 150.000 | | 69.400 | ................*X | 69.400 | | 50.000 | ...........*****X | 50.000 | | 9.700 | .********..........**X | 50.000 | | 40.300 | .........****X | 40.300 | | 0.000 | **************************************************************************************************************************************************************************************************************************************************************************************************..........***.....*******X | 59.700 | | 40.300 | .........*****X | 40.300 | | 0.000 | **..******..........*X | 50.000 | | 40.300 | .........*****X | 40.300 | | 0.000 | ***.....*****....................**X | 100.000 | | 80.600 | ...................*****X | 80.600 | | 0.000 | ****************....................***.....****X | 100.000 | | 19.400 | ....**X | 19.400 | | 0.000 | ******************************************************************************************************************************************************************************************************************************************************..................***........******.......................................**.....*****X | 274.400 | | 113.200 | ............................*X | 113.200 | | 74.400 | ..................*****X | 74.400 | | 50.000 | ...........*******X | 50.000 | | 9.700 | .**X | 9.700 | | 0.000 | ******************************************************************************************************************************************************************************************************..******..........*X | 50.000 | | 40.300 | .........********..*****X | 50.000 | | 9.700 | .**X | 9.700 | | 0.000 | *****..........***..******.....................**.....*****X | 150.000 | | 109.700 | ...........................********...............................................................**.........*****X | 387.900 | | 65.500 | ...............**X | 65.500 | | 0.000 | ***********************************************************************************************************************************************************************************...............................**.......*****X | 150.000 | | 29.100 | ......**X | 29.100 | | 0.000 | ******..........**..*****X | 50.000 | | 9.700 | .*X | 9.700 | | 0.000 | *****************..******...............................**............******...............................*X | 297.400 | | 283.200 | ........................................................................*****X | 283.200 | | 137.900 | ..................................**X | 137.900 | | 106.200 | ..........................****X | 106.200 | | 34.100 | .......*********....*X | 50.000 | | 40.300 | .........*****X | 40.300 | | 0.000 | ************************************************************************************************************************************************************.......................................***............*****X | 201.600 | | 96.900 | .......................*X | 96.900 | | 58.100 | .............*****X | 58.100 | | 50.000 | ...........*X | 50.000 | | 40.300 | .........*****X | 40.300 | | 0.000 | ********....................***..............******....................................................*X | 334.700 | | 316.800 | ................................................................................*****X | 316.800 | | 34.700 | .......*X | 34.700 | | 0.000 | ********************************************************************************************************************************X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 09:25:32,250 | test | INFO | pool-7-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..........................................................**.......................******...................**X | 28368.300 | | 26063.600 | ..........................................................................................*****X | 26063.600 | | 19260.100 | ..................................................................*X | 19260.100 | | 19236.500 | ..................................................................*****X | 19236.500 | | 16801.900 | ..........................................................*X | 16801.900 | | 14914.200 | ...................................................*****X | 14914.200 | | 4718.100 | ...............**X | 4718.100 | | 2559.500 | ........********....******.........**.******......***.****X | 8530.800 | | 7601.000 | .........................***.******.......**..******........*X | 12494.900 | | 12288.200 | ..........................................******..**....******............**X | 17453.800 | | 17399.900 | ............................................................****X | 17399.900 | | 13460.000 | ..............................................**X | 13460.000 | | 12548.100 | ...........................................******....**...*****X | 14733.400 | | 13342.900 | ..............................................*X | 13342.900 | | 12437.500 | ..........................................*****X | 12437.500 | | 11601.200 | .......................................*X | 11601.200 | | 11509.800 | .......................................*****X | 11509.800 | | 11151.600 | ......................................***.*****........................***....*****X | 19455.300 | | 12935.200 | ............................................*X | 12935.200 | | 10671.500 | ....................................*****X | 10671.500 | | 8918.200 | ..............................**.******.........**...******.**X | 12949.300 | | 12922.700 | ............................................****X | 12922.700 | | 11890.100 | ........................................**X | 11890.100 | | 11479.400 | .......................................******.**.*****X | 12091.100 | | 8119.300 | ...........................*X | 8119.300 | | 6980.600 | .......................******.......***..*****..........***..*****X | 12793.600 | | 12564.500 | ...........................................********........**.*****X | 15149.400 | | 10730.300 | ....................................**X | 10730.300 | | 9507.300 | ................................*****.......***.....******............*******X | 16424.200 | | 14301.900 | .................................................*X | 14301.900 | | 13927.500 | ................................................*****X | 13927.500 | | 13709.200 | ...............................................*X | 13709.200 | | 13586.400 | ..............................................*****X | 13586.400 | | 10930.200 | .....................................**X | 10930.200 | | 10565.600 | ....................................*****.**X | 10937.600 | | 10510.400 | ....................................*****X | 10510.400 | | 8680.600 | .............................**.******.............**.*****X | 13031.600 | | 9181.700 | ...............................*X | 9181.700 | | 8408.600 | ............................*****X | 8408.600 | | 7400.800 | .........................**X | 7400.800 | | 7212.300 | ........................******...*******X | 8170.100 | | 7947.400 | ...........................********.**.******........*******X | 11005.900 | | 7960.600 | ...........................**X | 7960.600 | | 7609.500 | .........................*****X | 7609.500 | | 6701.600 | ......................*X | 6701.600 | | 6076.800 | ....................******...**...******.......*******X | 9903.100 | | 8512.900 | .............................**X | 8512.900 | | 7883.600 | ..........................*****X | 7883.600 | | 6685.400 | ......................********......*******X | 8418.800 | | 8120.400 | ...........................***.*****......***.*****X | 10282.700 | | 9222.500 | ...............................*X | 9222.500 | | 8530.000 | .............................*****X | 8530.000 | | 8117.800 | ...........................**.*****X | 8360.100 | | 7462.100 | .........................**X | 7462.100 | | 7119.000 | ........................*****.***.******...*X | 8697.400 | | 8572.000 | .............................******.**..******...............**.*****X | 14087.900 | | 12318.000 | ..........................................**X | 12318.000 | | 11941.400 | .........................................*****.......***..*****X | 14644.200 | | 14304.500 | .................................................*X | 14304.500 | | 14245.300 | .................................................*****X | 14245.300 | | 12028.600 | .........................................*X | 12028.600 | | 11428.900 | .......................................*****X | 11428.900 | | 9293.000 | ...............................*X | 9293.000 | | 8445.400 | ............................*****X | 8445.400 | | 5723.300 | ...................**X | 5723.300 | | 5305.000 | .................*****.*....**..******.....**..******.............**...*****X | 13662.500 | | 11385.900 | .......................................**X | 11385.900 | | 10683.100 | ....................................*****..***..*****X | 11642.900 | | 10329.800 | ...................................*X | 10329.800 | | 9376.200 | ................................******.*X | 9774.100 | | 9606.300 | ................................*****X | 9606.300 | | 6643.800 | ......................**X | 6643.800 | | 6521.300 | .....................*****....***..******.....*X | 9542.500 | | 9331.100 | ...............................*****X | 9331.100 | | 9232.300 | ...............................**..*****X | 9682.400 | | 8787.600 | .............................*X | 8787.600 | | 8353.900 | ............................******.***..*****...**X | 10096.600 | | 10040.600 | ..................................******......**...******.....**.******..*X | 14993.400 | | 14404.600 | .................................................*****X | 14404.600 | | 10927.400 | .....................................**X | 10927.400 | | 10353.900 | ...................................****X | 10353.900 | | 7126.600 | ........................**X | 7126.600 | | 6704.200 | ......................******......*******X | 8484.400 | | 7024.400 | .......................*X | 7024.400 | | 6636.500 | ......................******..***.*****.....***.******......**.******..**.*******X | 12127.900 | | 11976.900 | .........................................******.********X | 12309.400 | | 10053.600 | ..................................*X | 10053.600 | | 9544.100 | ................................********.******...**.*********.*****.**X | 11447.100 | | 11341.400 | ......................................*****X | 11341.400 | | 10094.400 | ..................................*X | 10094.400 | | 9467.800 | ................................*****X | 9467.800 | | 7633.500 | .........................*X | 7633.500 | | 7606.100 | .........................*****X | 7606.100 | | 7454.100 | .........................********..........***..******.*******X | 11231.100 | | 10995.300 | .....................................*X | 10995.300 | | 10626.100 | ....................................*****X | 10626.100 | | 9602.900 | ................................**.******...........*******X | 12862.100 | | 9639.300 | ................................**X | 9639.300 | | 9318.300 | ...............................*****X | 9318.300 | | 9032.200 | ..............................*X | 9032.200 | | 8233.400 | ............................********...******.....*X | 10540.300 | | 10298.400 | ...................................*****X | 10298.400 | | 6946.400 | .......................**X | 6946.400 | | 6491.600 | .....................*****.............***...*****X | 11040.200 | | 9598.400 | ................................*X | 9598.400 | | 9168.100 | ...............................******............**...*****X | 13357.400 | | 12699.400 | ...........................................*X | 12699.400 | | 12580.800 | ...........................................*****X | 12580.800 | | 9815.200 | .................................**X | 9815.200 | | 8393.700 | ............................****X | 8393.700 | | 5714.500 | ...................***.******.......**.******.********........**....******.**X | 12252.400 | | 12157.300 | .........................................*****X | 12157.300 | | 9644.900 | ................................*X | 9644.900 | | 7915.300 | ..........................******..**.....******....*X | 10790.000 | | 10378.100 | ...................................*****X | 10378.100 | | 6432.700 | .....................**X | 6432.700 | | 5420.600 | ..................*****.................***.....*****X | 11819.800 | | 10583.000 | ....................................*X | 10583.000 | | 10236.700 | ...................................*****X | 10236.700 | | 8787.800 | .............................*X | 8787.800 | | 7896.300 | ..........................*****X | 7896.300 | | 6852.400 | .......................***..*****.....********X | 9046.200 | | 8996.100 | ..............................*X | 8996.100 | | 8546.200 | .............................******..**..*****X | 9784.300 | | 9471.700 | ................................********...*******X | 10356.600 | | 9582.000 | ................................**X | 9582.000 | | 8529.900 | .............................****X | 8529.900 | | 7720.900 | ..........................***..*****X | 8277.400 | | 7392.100 | .........................*X | 7392.100 | | 7244.700 | ........................******.......**..******....**.*****X | 11331.000 | | 8632.600 | .............................**X | 8632.600 | | 7323.400 | ........................*****X | 7323.400 | | 6748.700 | ......................**..*****X | 7272.000 | | 6307.600 | .....................*X | 6307.600 | | 6094.500 | ....................******..***********.******.....*******X | 8496.000 | | 8220.600 | ...........................*X | 8220.600 | | 8082.800 | ...........................******.......***.****X | 10331.300 | | 8073.200 | ...........................**X | 8073.200 | | 7867.700 | ..........................******.*X | 8043.100 | | 7781.400 | ..........................******.....**..*****X | 9663.900 | | 9096.800 | ...............................*********..**.******..*******X | 10695.200 | | 10410.700 | ...................................*X | 10410.700 | | 9696.900 | .................................*****X | 9696.900 | | 5654.300 | ..................**X | 5654.300 | | 5210.100 | .................******..........**....******...........**..*****X | 12973.700 | | 10383.100 | ...................................*X | 10383.100 | | 9051.200 | ..............................*****X | 9051.200 | | 6940.700 | .......................***..******.....................**.*****X | 13838.000 | | 3611.500 | ...........*X | 3611.500 | | 2253.000 | ......******........................***.....****X | 10334.300 | | 10214.300 | ...................................********X | 10371.800 | | 7268.400 | ........................*X | 7268.400 | | 5638.500 | ..................******....**....******........*******X | 10187.800 | | 7757.900 | ..........................*X | 7757.900 | | 7247.200 | ........................******..***.*******X | 8146.800 | | 8073.400 | ...........................******.....**.*****X | 9885.300 | | 8383.600 | ............................*X | 8383.600 | | 7454.400 | .........................******..***.*****X | 8420.300 | | 5494.200 | ..................*X | 5494.200 | | 4518.600 | ..............******..................**......********X | 11336.400 | | 10360.800 | ...................................****X | 10360.800 | | 7866.600 | ..........................*********.......**....*******X | 10910.400 | | 10152.400 | ..................................*********.*****.***.*******X | 11011.800 | | 10655.300 | ....................................*****X | 10655.300 | | 8693.500 | .............................*X | 8693.500 | | 8295.700 | ............................*****X | 8295.700 | | 7368.300 | ........................**X | 7368.300 | | 7255.200 | ........................*****X | 7255.200 | | 6456.000 | .....................*X | 6456.000 | | 6342.600 | .....................*****X | 6342.600 | | 6001.100 | ....................*X | 6001.100 | | 5762.200 | ...................******..**.*****X | 6602.000 | | 6360.200 | .....................**X | 6360.200 | | 6275.900 | .....................******........**..*****X | 9096.100 | | 6321.500 | .....................**.******.......**..******.............**X | 12874.300 | | 12745.300 | ...........................................*****X | 12745.300 | | 9718.600 | .................................*X | 9718.600 | | 9321.500 | ...............................*****X | 9321.500 | | 9038.600 | ..............................*X | 9038.600 | | 8670.600 | .............................*****X | 8670.600 | | 6847.200 | .......................**X | 6847.200 | | 6753.000 | ......................*****....***.******...*X | 8942.400 | | 8580.600 | .............................*****X | 8580.600 | | 6755.000 | ......................*X | 6755.000 | | 6655.800 | ......................*****X | 6655.800 | | 6361.300 | .....................********X | 6467.700 | | 6456.900 | .....................********..........**...******..*X | 10551.900 | | 10470.400 | ...................................******.....***.****X | 12061.400 | | 10623.100 | ....................................**X | 10623.100 | | 10359.000 | ...................................*****X | 10359.000 | | 9721.800 | .................................*X | 9721.800 | | 9497.600 | ................................*****X | 9497.600 | | 8549.900 | .............................*X | 8549.900 | | 8405.600 | ............................******.........**...*****X | 11664.700 | | 10532.000 | ....................................**X | 10532.000 | | 9936.000 | ..................................****X | 9936.000 | | 9402.600 | ................................X | 9402.600 | | 8912.000 | ..............................**.******...*X | 10074.400 | | 9836.000 | .................................*****X | 9836.000 | | 8524.700 | .............................*X | 8524.700 | | 8403.600 | ............................******.........***...******........*X | 13983.000 | | 13865.200 | ...............................................*****X | 13865.200 | | 9514.500 | ................................*X | 9514.500 | | 8850.700 | ..............................******....**.*****X | 10477.700 | | 8244.400 | ............................**X | 8244.400 | | 7383.300 | .........................******....**.....******...............**.*****X | 14597.500 | | 12515.700 | ...........................................*X | 12515.700 | | 12111.100 | .........................................******.......***.*****X | 14403.800 | | 13249.700 | .............................................*X | 13249.700 | | 13030.500 | ............................................******.**.*****X | 13580.600 | | 12883.300 | ............................................**X | 12883.300 | | 12510.300 | ...........................................*****..........***..*****X | 16014.700 | | 14275.200 | .................................................*X | 14275.200 | | 13871.800 | ...............................................*****X | 13871.800 | | 12631.500 | ...........................................*X | 12631.500 | | 12452.300 | ..........................................******.....*******X | 13771.000 | | 11625.000 | .......................................**X | 11625.000 | | 10822.900 | .....................................*****X | 10822.900 | | 9110.500 | ...............................*X | 9110.500 | | 8790.100 | .............................*****X | 8790.100 | | 6944.200 | .......................*X | 6944.200 | | 6785.100 | ......................******.**X | 6857.500 | | 6669.400 | ......................******.**.*****X | 7121.900 | | 6286.200 | .....................*X | 6286.200 | | 6125.600 | ....................*****X | 6125.600 | | 5583.600 | ..................**X | 5583.600 | | 5434.600 | ..................****X | 5434.600 | | 5060.100 | ................***.......******.................*X | 11894.700 | | 11301.400 | ......................................******.....**.******....***.****X | 14225.800 | | 13462.300 | ..............................................**X | 13462.300 | | 12815.400 | ............................................*****X | 12815.400 | | 11519.100 | .......................................*X | 11519.100 | | 11344.500 | ......................................*****X | 11344.500 | | 9590.400 | ................................*X | 9590.400 | | 8031.700 | ...........................*****X | 8031.700 | | 7825.900 | ..........................***.....*****..*........**.*****X | 12457.700 | | 11674.600 | ........................................*X | 11674.600 | | 11435.600 | .......................................******....**..******.**X | 13580.700 | | 13344.100 | ..............................................****X | 13344.100 | | 10349.900 | ...................................**X | 10349.900 | | 10142.300 | ..................................******........**..*****X | 12847.800 | | 10967.500 | .....................................*X | 10967.500 | | 10261.100 | ...................................*****X | 10261.100 | | 10110.200 | ..................................**.******............**....*****X | 14841.600 | | 12257.300 | ..........................................**X | 12257.300 | | 11435.300 | .......................................******......**.*****X | 13412.900 | | 11797.400 | ........................................*X | 11797.400 | | 11783.300 | ........................................******.**.*****X | 12286.900 | | 10516.700 | ....................................**X | 10516.700 | | 9810.700 | .................................******.....**.******....*******X | 12578.300 | | 8606.500 | .............................*X | 8606.500 | | 7785.800 | ..........................******......***...*****X | 10269.100 | | 9848.000 | .................................*X | 9848.000 | | 9577.800 | ................................******.......**..******.....**.*****X | 13701.200 | | 12758.400 | ...........................................**X | 12758.400 | | 12563.200 | ...........................................*****X | 12563.200 | | 11429.900 | .......................................*X | 11429.900 | | 10890.600 | .....................................*****X | 10890.600 | | 10296.300 | ...................................**..******........*******X | 13155.900 | | 12698.700 | ...........................................**X | 12698.700 | | 12308.600 | ..........................................*****X | 12308.600 | | 8981.600 | ..............................*X | 8981.600 | | 8447.700 | ............................******.....**..*****X | 10332.400 | | 8114.600 | ...........................**X | 8114.600 | | 7010.600 | .......................*****X | 7010.600 | | 5779.300 | ...................********.....**..*****X | 7688.500 | | 5914.700 | ...................**X | 5914.700 | | 5259.000 | .................*****........***...******.............**....******......*X | 14946.000 | | 14400.600 | .................................................*****X | 14400.600 | | 12315.300 | ..........................................*X | 12315.300 | | 12214.400 | ..........................................*****X | 12214.400 | | 9020.200 | ..............................**X | 9020.200 | | 8441.000 | ............................******....**.******.*X | 9996.700 | | 9806.700 | .................................*****X | 9806.700 | | 7852.300 | ..........................**X | 7852.300 | | 7481.800 | .........................****X | 7481.800 | | 7067.500 | .......................***.******...........**...******.*X | 11628.800 | | 11608.100 | .......................................******......***..******.....*X | 15040.000 | | 14960.600 | ...................................................*****X | 14960.600 | | 13659.300 | ...............................................*X | 13659.300 | | 13606.200 | ..............................................******..*X | 14039.900 | | 13825.500 | ...............................................*****X | 13825.500 | | 11478.900 | .......................................**X | 11478.900 | | 11196.300 | ......................................*****......***.******.....*******X | 14649.400 | | 13881.700 | ...............................................*X | 13881.700 | | 13684.400 | ...............................................*****X | 13684.400 | | 9534.700 | ................................**X | 9534.700 | | 8996.400 | ..............................*****..............***...******.*X | 14099.100 | | 13615.200 | ..............................................*****X | 13615.200 | | 10623.800 | ....................................********..*X | 11199.300 | | 11036.800 | .....................................******.........***.*****X | 13701.500 | | 10781.400 | .....................................**.******............*X | 14645.900 | | 14624.100 | ..................................................*****X | 14624.100 | | 14126.100 | ................................................*X | 14126.100 | | 13978.100 | ................................................*****X | 13978.100 | | 10186.200 | ..................................**X | 10186.200 | | 9734.800 | .................................******......**..******.......**.*****X | 14371.800 | | 13092.700 | .............................................********...**X | 14011.800 | | 13645.800 | ...............................................******....**...*****X | 15605.500 | | 13313.300 | .............................................*X | 13313.300 | | 12285.200 | ..........................................*****X | 12285.200 | | 9750.500 | .................................*X | 9750.500 | | 9420.300 | ................................******.***.*****X | 10133.200 | | 8844.200 | ..............................*X | 8844.200 | | 8243.900 | ............................*****X | 8243.900 | | 7302.000 | ........................*X | 7302.000 | | 7272.100 | ........................******.......**..******.............***..****X | 14146.700 | | 12354.600 | ..........................................**X | 12354.600 | | 12163.500 | .........................................******........*X | 14434.100 | | 14259.400 | .................................................*****X | 14259.400 | | 10146.600 | ..................................*X | 10146.600 | | 9643.800 | ................................******................**....****X | 15176.700 | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 09:25:33,520 | test | INFO | pool-7-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 15176.700 | ...................................................................................................X | 15176.700 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 09:25:33,635 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 26.8392196863 | 11.45 GiB | 8.48 GiB | 52.50 MiB / 3.50 GiB | 6489378 / 13023026 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 33.8231535906 | 11.45 GiB | 8.42 GiB | 24.25 MiB / 3.50 GiB | 6512810 / 13011130 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 46.2248429689 | 11.45 GiB | 7.90 GiB | 28.14 MiB / 3.50 GiB | 6501837 / 13010000 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 26.7510224949 | 11.45 GiB | 8.67 GiB | 0.0 Byte / 3.50 GiB | 6484099 / 12941153 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 33.252520097 | 11.45 GiB | 8.29 GiB | 0.0 Byte / 3.50 GiB | 6503917 / 12987491 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 09:25:51,003 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 09:26:05,415 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 09:26:05,415 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 09:26:16,394 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 09:26:16,394 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 09:26:28,618 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 09:26:28,619 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 09:26:46,885 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 242.92 MiB | 315.31 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 520.57 MiB | 565.40 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.63 GiB | 49.78 GiB | 22.8881320031 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 09:26:46,887 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 09:26:51,308 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_recovery 2022-02-02 09:26:51,309 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_recovery 2022-02-02 09:26:51,309 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:670] failing over nodes [ip:172.23.105.36 port:8091 ssh_username:root, ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 09:27:20,599 | test | INFO | pool-7-thread-1 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 29.1529998779 seconds to complete 2022-02-02 09:27:20,601 | test | INFO | pool-7-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 09:32:46,884 | test | INFO | pool-7-thread-20 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 21.1519999504 seconds to complete 2022-02-02 09:32:46,890 | test | INFO | pool-7-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 09:37:51,891 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 09:37:51,990 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 09:37:51,999 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 09:50:20,078 | test | INFO | pool-7-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******....................................................................................................*******X | 436.000 | | 82.000 | .................********X | 82.000 | | 24.000 | ....********.................*******X | 96.000 | | 68.000 | ..............********X | 68.000 | | 66.000 | ..............********.*******X | 72.000 | | 58.000 | ............********X | 58.000 | | 8.000 | ********..************************......................*******X | 110.000 | | 60.000 | ............*******X | 60.000 | | 8.000 | ********...............*******X | 72.000 | | 36.000 | .......********.....********..*********........*******X | 102.000 | | 60.000 | ............*******X | 60.000 | | 36.000 | .......********.********X | 42.000 | | 36.000 | .......********........*******X | 72.000 | | 58.000 | ............****************.......*******X | 88.000 | | 62.000 | .............*******X | 62.000 | | 38.000 | .......*********........*******X | 70.000 | | 52.000 | ..........*******X | 52.000 | | 42.000 | ........********.....*******X | 62.000 | | 52.000 | ..........*******X | 52.000 | | 38.000 | .......********X | 38.000 | | 34.000 | ......*******X | 34.000 | | 26.000 | ....********................********X | 92.000 | | 34.000 | ......********.....*******X | 56.000 | | 36.000 | .......***************X | 38.000 | | 10.000 | .*******X | 10.000 | | 0.000 | *********..********......................*******X | 108.000 | | 44.000 | .........********X | 44.000 | | 16.000 | ..********......*******X | 42.000 | | 38.000 | .......*******X | 38.000 | | 26.000 | ....****************X | 26.000 | | 8.000 | ********......*******X | 34.000 | | 28.000 | .....********...********..*******X | 50.000 | | 8.000 | *********....*******X | 24.000 | | 16.000 | ..********.*******X | 18.000 | | 16.000 | ..*********..........*******X | 60.000 | | 48.000 | ..........*******X | 48.000 | | 8.000 | *******X | 8.000 | | 0.000 | *********...*******X | 16.000 | | 8.000 | ********....********X | 24.000 | | 16.000 | ..*******X | 16.000 | | 0.000 | *********************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 09:50:20,299 | test | INFO | pool-7-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******....................................................................................................*******X | 288.000 | | 56.000 | ..................*******X | 56.000 | | 8.000 | .*********................................................*******X | 144.000 | | 122.000 | .........................................*******X | 122.000 | | 82.000 | ...........................*******X | 82.000 | | 66.000 | .....................********......********X | 82.000 | | 50.000 | ................X | 50.000 | | 50.000 | ................******X | 50.000 | | 26.000 | ........********.....*******X | 42.000 | | 24.000 | .......********..........................*******X | 98.000 | | 40.000 | ............*******X | 40.000 | | 34.000 | ..........********....*******X | 44.000 | | 24.000 | .......*********.........****X | 50.000 | | 50.000 | ................*X | 50.000 | | 50.000 | ................*...********.......********.............********X | 118.000 | | 52.000 | .................********.........*******X | 80.000 | | 26.000 | ........********.........********....................*******X | 110.000 | | 64.000 | .....................********X | 64.000 | | 60.000 | ...................*******X | 60.000 | | 42.000 | .............*******X | 42.000 | | 18.000 | .....********......*******X | 36.000 | | 26.000 | ........********...........********X | 58.000 | | 54.000 | .................********.*******X | 56.000 | | 18.000 | .....********............*******X | 54.000 | | 54.000 | .................*********.........*******X | 80.000 | | 55.500 | ..................*******X | 55.500 | | 38.500 | ............********..................*******X | 90.000 | | 28.000 | ........*******X | 28.000 | | 20.000 | .....********X | 20.000 | | 0.000 | ********...............*******X | 46.000 | | 26.000 | ........********...........********X | 60.000 | | 24.000 | .......********.........*******X | 50.000 | | 40.000 | ............*******X | 40.000 | | 8.000 | .*******X | 8.000 | | 0.000 | ********.*.********..........*******X | 36.000 | | 16.000 | ....****************.............*******X | 52.000 | | 14.100 | ...********...........********X | 45.900 | | 16.000 | ....*******X | 16.000 | | 8.000 | .********...********........*********............*******X | 74.000 | | 0.000 | ****************......********X | 18.000 | | 0.000 | ********......********.....********X | 32.000 | | 0.000 | ********..********....*****X | 17.400 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 09:50:21,427 | test | INFO | pool-7-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.........********........................********.......................*********..............*******X | 38330.300 | | 37310.100 | ...................................................................********.......*******X | 41355.000 | | 38032.600 | ....................................................................*******X | 38032.600 | | 20942.700 | .....................................*********.********....********.....********......********................********.*******X | 39033.500 | | 34006.700 | .............................................................********......................................*******X | 54496.700 | | 32114.500 | .........................................................*******X | 32114.500 | | 21969.400 | .......................................********......................********......*....******X | 39469.200 | | 36494.000 | .................................................................X | 36494.000 | | 30689.600 | .......................................................********........***X | 35205.100 | | 35205.100 | ...............................................................**X | 35205.100 | | 35205.100 | ...............................................................X | 35205.100 | | 30174.200 | ......................................................********..********.......********..............*******X | 42623.900 | | 30962.700 | .......................................................********..........********......................*******X | 48181.600 | | 30469.800 | ......................................................********......................*******X | 42012.700 | | 32980.700 | ...........................................................*********...********...........*******X | 40425.700 | | 39722.000 | .......................................................................*******X | 39722.000 | | 35493.100 | ................................................................********..********X | 36534.200 | | 29297.600 | ....................................................*******X | 29297.600 | | 24737.000 | ............................................********..............********....*******X | 34866.800 | | 30500.800 | ......................................................*******X | 30500.800 | | 22881.500 | ........................................*******X | 22881.500 | | 18738.300 | .................................********.....********..............********X | 29056.600 | | 25310.600 | .............................................********.*******X | 25801.400 | | 25390.900 | .............................................********...............*******X | 33635.000 | | 28116.100 | ..................................................********X | 28116.100 | | 22887.800 | ........................................********..*******X | 23632.400 | | 22114.800 | .......................................********.............********.......*********...*******X | 34408.400 | | 32667.900 | ..........................................................*******X | 32667.900 | | 24336.900 | ...........................................********.....********X | 27149.000 | | 26310.900 | ...............................................********...*******X | 28064.400 | | 24241.200 | ...........................................********......********...********X | 28953.200 | | 23713.600 | ..........................................********.....*******X | 26526.600 | | 26102.700 | ..............................................*******X | 26102.700 | | 21002.300 | .....................................*********.....*******X | 23741.100 | | 22589.600 | ........................................*******X | 22589.600 | | 20776.500 | .....................................*******X | 20776.500 | | 19170.100 | ..................................*********...........*****X | 25114.700 | +-----------+---------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 09:50:22,674 | test | INFO | pool-7-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 17.400 | ...................................................................................................X | 17.400 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 09:50:22,690 | test | INFO | pool-7-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 25114.700 | ...................................................................................................X | 25114.700 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 09:50:22,796 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 09:50:22,796 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 09:50:22,815 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 09:50:22,815 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 09:50:39,005 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 09:50:39,010 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 09:50:39,233 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 0.613343347102 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 19.6318244862 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 7.38879115356 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.350789275871 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 11.3839566805 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 09:50:39,250 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 09:50:49,276 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 09:50:59,562 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 26.67 2022-02-02 09:51:09,605 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:51:19,647 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:51:29,684 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:51:39,724 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:51:49,772 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:51:59,811 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:52:09,845 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:52:19,882 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:52:29,920 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:52:39,959 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:52:50,003 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:53:00,048 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:53:10,098 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:53:20,142 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:53:30,177 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:53:40,216 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:53:50,266 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:54:00,318 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:54:10,368 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:54:20,410 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:54:30,460 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:54:40,503 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:54:50,543 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:55:00,585 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:55:10,621 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:55:20,661 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:55:30,701 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:55:40,737 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:55:50,773 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:56:00,818 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:56:10,861 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:56:20,907 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:56:30,948 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:56:40,984 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:56:51,035 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:57:01,079 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:57:11,114 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:57:21,158 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:57:31,200 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:57:41,244 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:57:51,301 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:58:01,345 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:58:11,387 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:58:21,433 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:58:31,476 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 09:58:41,516 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.38 2022-02-02 09:58:51,549 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.38 2022-02-02 09:59:01,586 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.38 2022-02-02 09:59:11,628 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.38 2022-02-02 09:59:21,665 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.38 2022-02-02 09:59:31,716 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.52 2022-02-02 09:59:41,760 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.71 2022-02-02 09:59:51,796 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.9 2022-02-02 10:00:01,832 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.9 2022-02-02 10:00:11,881 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.9 2022-02-02 10:00:21,915 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.94 2022-02-02 10:00:31,947 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.94 2022-02-02 10:00:41,983 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.94 2022-02-02 10:00:52,016 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 53.99 2022-02-02 10:01:02,055 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.04 2022-02-02 10:01:12,092 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.04 2022-02-02 10:01:22,134 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.08 2022-02-02 10:01:32,174 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:01:42,232 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:01:52,266 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:02:02,325 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:02:12,361 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:02:22,395 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:02:32,431 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:02:42,469 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:02:52,505 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:03:02,540 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:03:12,579 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:03:22,617 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:03:32,661 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:03:42,696 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:03:52,733 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:04:02,788 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:04:09,582 | test | INFO | pool-7-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********...................................................................................................***.****X | 275.700 | | 27.600 | .........***..*****.....***...........******......................*X | 138.000 | | 105.200 | .....................................*****X | 105.200 | | 3.400 | **...******..........................**...*****X | 92.700 | | 73.500 | .........................**X | 73.500 | | 56.500 | ...................*****......***........*****X | 95.100 | | 67.100 | .......................*X | 67.100 | | 53.000 | ..................*****X | 53.000 | | 33.800 | ...........*X | 33.800 | | 28.700 | .........*****X | 28.700 | | 11.000 | ..**.******........................**...*****X | 86.100 | | 38.900 | .............**X | 38.900 | | 30.400 | ..........*****..***.*****...X | 47.100 | | 47.100 | ................***X | 47.600 | | 47.600 | ................**X | 47.600 | | 22.500 | .......**X | 22.500 | | 15.700 | ....****X | 15.700 | | 11.300 | ...********........***.****X | 36.100 | | 29.400 | .........X | 29.400 | | 22.800 | .......**.******..*X | 32.100 | | 27.000 | ........******.**.....*****..........**X | 71.200 | | 65.200 | ......................*****X | 65.200 | | 41.900 | ..............*X | 41.900 | | 36.100 | ............*****.***...*****.**X | 50.300 | | 45.900 | ...............****X | 45.900 | | 38.500 | ............**X | 38.500 | | 38.000 | ............*****.......***.****X | 59.500 | | 56.600 | ...................***..****X | 60.700 | | 53.200 | ..................**X | 53.200 | | 45.800 | ...............****X | 45.800 | | 20.800 | ......*********.................*X | 66.200 | | 64.500 | ......................*****X | 64.500 | | 40.800 | .............**.*****X | 43.700 | | 30.400 | ..........*X | 30.400 | | 26.700 | ........******.....**.******.....*******X | 57.500 | | 44.300 | ...............**X | 44.300 | | 41.600 | ..............****X | 41.600 | | 37.100 | ............**X | 37.100 | | 36.800 | ............****X | 36.800 | | 29.500 | .........***.*****..............**X | 70.700 | | 70.600 | ........................****X | 70.600 | | 34.000 | ...........**X | 34.000 | | 26.600 | ........*****.***..*****.............***..****X | 74.600 | | 43.500 | ..............**X | 43.500 | | 42.600 | ..............*****......**X | 59.900 | | 55.200 | ...................*****.***.*****X | 62.800 | | 27.400 | ........*X | 27.400 | | 17.200 | .....******..**********..******...............**.*****X | 74.000 | | 21.200 | ......**X | 21.200 | | 11.800 | ...*****..**X | 19.200 | | 18.800 | .....*****.............***.....****X | 67.400 | | 63.300 | .....................**X | 63.300 | | 54.800 | ..................****X | 54.800 | | 16.700 | .....**X | 16.700 | | 9.600 | ..******.......**...******.*******X | 40.000 | | 30.000 | .........*X | 30.000 | | 28.600 | .........*****X | 28.600 | | 13.800 | ....**X | 13.800 | | 5.800 | .*************.................***.....****X | 66.800 | | 21.300 | ......**X | 21.300 | | 11.200 | ...******......**.*****X | 30.800 | | 20.800 | ......*X | 20.800 | | 13.200 | ...*****X | 13.200 | | 5.800 | .***...*****............***...*******X | 57.000 | | 49.400 | ................*****X | 49.400 | | 24.600 | .......*X | 24.600 | | 21.000 | ......******.........**...*****X | 53.400 | | 17.000 | .....**X | 17.000 | | 7.600 | .****X | 7.600 | | 1.800 | **X | 1.800 | | 0.000 | ******....**...******..........*X | 49.600 | | 47.800 | ................*****X | 47.800 | | 18.800 | .....*X | 18.800 | | 15.200 | ....*****X | 15.200 | | 9.400 | ..*X | 9.400 | | 7.600 | .********.******..*******X | 15.200 | | 9.400 | ..**X | 9.400 | | 5.800 | .******....**..*****X | 24.800 | | 11.600 | ...*X | 11.600 | | 7.600 | .*****X | 7.600 | | 1.800 | **X | 1.800 | | 0.000 | *X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 10:04:09,756 | test | INFO | pool-7-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********................................*******X | 38.000 | | 22.800 | ..................***********...................*****.....................*.........*X | 81.100 | | 60.700 | ..................................................*****X | 60.700 | | 1.700 | **....******..............................**..*****X | 43.900 | | 26.200 | .....................*X | 26.200 | | 19.400 | ...............******..............................**.............******.**X | 71.600 | | 67.000 | .......................................................*****X | 67.000 | | 36.000 | .............................*X | 36.000 | | 30.400 | ........................********.......******..................................................**.*****X | 99.600 | | 28.800 | .......................*X | 28.800 | | 13.500 | ..........*****X | 13.500 | | 7.600 | .....***.......*****.........................................***.****X | 66.000 | | 34.500 | ............................**X | 34.500 | | 26.000 | ....................****X | 26.000 | | 15.200 | ...........***..*****....................................***.........****X | 70.400 | | 63.000 | ....................................................**X | 63.000 | | 57.400 | ...............................................*****X | 57.400 | | 22.000 | .................*X | 22.000 | | 16.900 | .............******.....**......*****..........................***.*****X | 62.800 | | 45.100 | .....................................*X | 45.100 | | 32.700 | ..........................*****................................***.......................*****..................**X | 118.600 | | 107.400 | .........................................................................................****X | 107.400 | | 29.400 | .......................**X | 29.400 | | 19.100 | ...............*****......**X | 26.500 | | 22.600 | ..................*****................................***........****X | 71.000 | | 17.200 | .............**X | 17.200 | | 5.500 | ...****X | 5.500 | | 3.800 | ..***.........*****...............................................**X | 70.500 | | 63.200 | ....................................................*****........***..............****X | 89.800 | | 47.100 | ......................................**X | 47.100 | | 28.500 | .......................*****X | 28.500 | | 9.400 | ......**....******................................*X | 51.600 | | 51.500 | ..........................................*****X | 51.500 | | 33.800 | ...........................**......******.....*X | 46.400 | | 35.900 | .............................****X | 35.900 | | 32.800 | ..........................X | 32.800 | | 22.700 | ..................**........******...........................*X | 64.500 | | 59.400 | .................................................*****X | 59.400 | | 13.700 | ..........*X | 13.700 | | 7.600 | .....******.............................**........*****X | 51.600 | | 19.200 | ...............*X | 19.200 | | 9.600 | .......******.....................................**..............******.....................**.*****X | 97.100 | | 60.600 | ..................................................*X | 60.600 | | 55.800 | ..............................................*****X | 55.800 | | 34.900 | ............................*X | 34.900 | | 21.400 | .................*****X | 21.400 | | 9.600 | .......*X | 9.600 | | 7.400 | .....******..........***......*****.................................***.......*****X | 74.600 | | 15.800 | ............*X | 15.800 | | 4.000 | ..******.................**.******.................**.........*******X | 55.800 | | 49.200 | ........................................*****X | 49.200 | | 27.000 | .....................*X | 27.000 | | 24.800 | ...................******..............***....*****..........**......*****X | 65.200 | | 52.000 | ..........................................**X | 52.000 | | 40.000 | ................................****X | 40.000 | | 13.600 | ..........**X | 13.600 | | 11.800 | ........******........................**...******................**.......*****X | 70.800 | | 23.800 | ...................*X | 23.800 | | 11.300 | ........******.........................***.............****X | 56.800 | | 50.300 | .........................................**X | 50.300 | | 32.900 | ..........................****X | 32.900 | | 28.000 | ......................X | 28.000 | | 13.400 | ..........**.....******..........................................**............*****X | 83.200 | | 73.200 | ............................................................*X | 73.200 | | 57.400 | ...............................................*****X | 57.400 | | 19.400 | ...............*X | 19.400 | | 17.200 | .............******........................***...........****X | 58.800 | | 51.400 | ..........................................**X | 51.400 | | 43.800 | ...................................*****X | 43.800 | | 29.600 | .......................*X | 29.600 | | 25.000 | ....................*****X | 25.000 | | 21.800 | .................**....******.....*X | 32.200 | | 29.000 | .......................*****X | 29.000 | | 24.800 | ...................*X | 24.800 | | 22.600 | ..................******..................**.......*****X | 53.200 | | 24.200 | ...................*X | 24.200 | | 15.200 | ...........******..........***.......*****.**X | 36.000 | | 28.400 | ......................*****X | 28.400 | | 21.000 | ................**.....******..........*X | 38.000 | | 34.400 | ............................*****X | 34.400 | | 5.400 | ...*X | 5.400 | | 0.000 | **X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 10:04:10,367 | test | INFO | pool-7-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ********..............................********................********......********.........................********X | 35253.800 | | 23584.800 | ..................................................********......................*******X | 33655.200 | | 28075.400 | ............................................................********........*******X | 31529.000 | | 30966.000 | ..................................................................*******X | 30966.000 | | 24897.400 | .....................................................X | 24897.400 | | 24897.400 | .....................................................****X | 24897.400 | | 24897.400 | .....................................................**.....********..........********X | 31590.200 | | 27051.600 | ..........................................................*******X | 27051.600 | | 26330.900 | ........................................................********...................*******X | 34850.200 | | 33959.100 | .........................................................................***************X | 34201.500 | | 29776.800 | ................................................................*******X | 29776.800 | | 25355.700 | ......................................................********.......********.......*******X | 31877.600 | | 30098.200 | ................................................................*******X | 30098.200 | | 28898.400 | ..............................................................********.....................................*******X | 45656.900 | | 43050.900 | .............................................................................................*******X | 43050.900 | | 36416.300 | ..............................................................................*******X | 36416.300 | | 30649.600 | ..................................................................*******X | 30649.600 | | 27571.000 | ...........................................................*******X | 27571.000 | | 27006.100 | ..........................................................********....********..*******X | 29752.800 | | 28753.100 | .............................................................********...........*X | 33476.400 | | 33476.400 | ........................................................................***X | 33476.400 | | 33476.400 | ........................................................................*X | 33476.400 | | 31578.000 | ....................................................................********X | 31578.000 | | 28225.100 | ............................................................********.............******X | 34156.500 | | 31352.700 | ...................................................................*******X | 31352.700 | | 28625.700 | .............................................................********.................********...*******X | 37550.400 | | 33450.500 | ........................................................................********...................********..*******X | 43011.200 | | 30352.100 | .................................................................*******X | 30352.100 | | 23442.000 | ..................................................********........*******X | 27223.200 | | 17289.000 | ....................................********.............*******X | 23154.100 | | 21723.200 | ..............................................*******X | 21723.200 | | 20213.600 | ...........................................********...............********.......*******X | 30270.400 | | 26477.300 | ........................................................********...........*******X | 31263.400 | | 26791.600 | .........................................................*******X | 26791.600 | | 24712.800 | .....................................................********.********...*******X | 26494.600 | | 21459.400 | ..............................................*******X | 21459.400 | | 19196.500 | .........................................********..............********X | 25827.600 | | 16985.900 | ....................................********........********..*******X | 21899.900 | | 18665.100 | .......................................********...........*******X | 23476.400 | | 20506.000 | ...........................................*******X | 20506.000 | | 19790.800 | ..........................................********.*...*******X | 21534.800 | | 20541.800 | ...........................................*******X | 20541.800 | | 16741.700 | ...................................********............*******X | 22359.800 | | 17918.600 | ......................................********..********..............*******X | 25343.700 | | 22640.400 | ................................................*******X | 22640.400 | | 16833.300 | ...................................********..........*****************....****X | 22900.100 | +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 10:04:11,671 | test | INFO | pool-7-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 22900.100 | ...................................................................................................X | 22900.100 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 10:04:12,826 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:04:22,859 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:04:32,892 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:04:42,924 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:04:52,960 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.23 2022-02-02 10:05:02,996 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:05:13,028 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:05:23,062 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:05:33,098 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:05:43,131 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:05:53,164 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:06:03,200 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:06:13,232 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:06:23,266 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:06:33,299 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 54.93 2022-02-02 10:06:43,346 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 55.62 2022-02-02 10:06:53,381 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 61.58 2022-02-02 10:07:03,417 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: running, progress: 75.8 2022-02-02 10:07:18,460 | test | INFO | pool-7-thread-6 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 10:07:18,480 | test | INFO | pool-7-thread-6 | [task:check:468] Rebalance completed with progress: 100% in 999.246000051 sec 2022-02-02 10:07:18,586 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 10:07:24,842 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 10:07:39,267 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 10:07:39,267 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 10:07:50,147 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 10:07:50,147 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 10:08:00,987 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 10:08:00,987 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 10:08:19,125 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 46000 | 9.77 GiB | 204.96 MiB | 307.60 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 4.88 GiB | 448.36 MiB | 577.42 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 30021250 | 2.50 GiB | 1.30 GiB | 55.72 GiB | 11.7999183911 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 10:08:19,614 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 10:08:19,614 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 10:08:26,069 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 10:08:26,071 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 10:08:32,530 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 17% 2022-02-02 10:08:32,530 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 6% 2022-02-02 10:09:26,918 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 10:09:27,332 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 10:09:27,332 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 10:09:57,532 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 10:09:57,969 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 10:09:57,970 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 10:10:28,075 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 10:10:28,471 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 10:10:28,473 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 10:13:29,780 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ce0c43f70a788a31bb3137470b662e7f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=76da884049736343a798165810e21bb9', u'status': u'notRunning'} - rebalance failed 2022-02-02 10:13:29,806 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 10:13:29,806 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643825608667L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:13:28.667Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 338c8022fe04e4baaafcb099ae630535'} 2022-02-02 10:13:29,808 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643825608662L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:13:28.662Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 10:13:29,808 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643825483901L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:11:23.901Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 10:13:29,808 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643825473149L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:11:13.149Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 10:13:29,808 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643825428659L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:10:28.659Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 338c8022fe04e4baaafcb099ae630535"} 2022-02-02 10:13:29,809 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643825428651L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T10:10:28.651Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 10:13:29,809 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643825421390L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:10:21.390Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 10:13:29,809 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643825370514L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:09:30.514Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything 2022-02-02 10:13:29,809 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643825337016L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:08:57.016Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} 2022-02-02 10:13:29,809 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643825337004L, u'shortText': u'message', u'serverTime': u'2022-02-02T10:08:57.004Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.33\' for deletion'} self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ce0c43f70a788a31bb3137470b662e7f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=76da884049736343a798165810e21bb9', u'status': u'notRunning'} - rebalance failed 2022-02-02 10:13:29,811 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ce0c43f70a788a31bb3137470b662e7f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=76da884049736343a798165810e21bb9', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 6364.107s OK During the test, Remote Connections: 256, Disconnections: 256 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_4 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=hard_failover_and_full_recovery_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_5', 'nodes_init': '5', 'GROUP': 'hard_failover_and_full_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 5, 'randomize_value': 'True'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 10:13:35,720 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:13:36,088 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:15:45,036 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.703999996 seconds to complete 2022-02-02 10:15:45,043 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 10:15:50,128 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x3dcc2c type=client open_count=1 channel=[id: 0x4cea7cd6, 0.0.0.0/0.0.0.0:46128] timeout=300.0>) 2022-02-02 10:15:53,680 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x3dcc2d type=client open_count=1 channel=[id: 0x1be7f36a, 0.0.0.0/0.0.0.0:56053] timeout=300.0>) 2022-02-02 10:15:55,691 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x3dcc2e type=client open_count=1 channel=[id: 0x4599271f, 0.0.0.0/0.0.0.0:35512] timeout=300.0>) 2022-02-02 10:15:57,701 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x3dcc2f type=client open_count=1 channel=[id: 0xa498059e, 0.0.0.0/0.0.0.0:35196] timeout=300.0>) 2022-02-02 10:15:59,711 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x3dcc30 type=client open_count=1 channel=[id: 0xad59b17e, 0.0.0.0/0.0.0.0:59390] timeout=300.0>) 2022-02-02 10:16:05,617 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x3dcc31 type=client open_count=1 channel=[id: 0xd6b8ee03, 0.0.0.0/0.0.0.0:59305] timeout=300.0>) 2022-02-02 10:16:07,627 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x3dcc32 type=client open_count=1 channel=[id: 0x5d8ff43a, 0.0.0.0/0.0.0.0:38602] timeout=300.0>) 2022-02-02 10:16:09,637 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x3dcc33 type=client open_count=1 channel=[id: 0x8ff3ffb2, 0.0.0.0/0.0.0.0:50855] timeout=300.0>) 2022-02-02 10:16:11,647 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x3dcc34 type=client open_count=1 channel=[id: 0xa0ddd686, 0.0.0.0/0.0.0.0:56432] timeout=300.0>) 2022-02-02 10:16:17,039 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x3dcc35 type=client open_count=1 channel=[id: 0xc47c65d1, 0.0.0.0/0.0.0.0:47177] timeout=300.0>) 2022-02-02 10:16:19,049 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x3dcc36 type=client open_count=1 channel=[id: 0x80ca64df, 0.0.0.0/0.0.0.0:59121] timeout=300.0>) 2022-02-02 10:16:21,059 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x3dcc37 type=client open_count=1 channel=[id: 0x0ff6fd70, 0.0.0.0/0.0.0.0:49380] timeout=300.0>) 2022-02-02 10:16:23,069 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x3dcc38 type=client open_count=1 channel=[id: 0x76b35ee5, 0.0.0.0/0.0.0.0:60971] timeout=300.0>) 2022-02-02 10:16:28,483 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x3dcc39 type=client open_count=1 channel=[id: 0xa742b2c3, 0.0.0.0/0.0.0.0:49545] timeout=300.0>) 2022-02-02 10:16:30,493 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x3dcc3a type=client open_count=1 channel=[id: 0xf498b580, 0.0.0.0/0.0.0.0:52651] timeout=300.0>) 2022-02-02 10:16:32,503 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x3dcc3b type=client open_count=1 channel=[id: 0x39b4c209, 0.0.0.0/0.0.0.0:39156] timeout=300.0>) 2022-02-02 10:16:34,512 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x3dcc3c type=client open_count=1 channel=[id: 0x5cc4588a, 0.0.0.0/0.0.0.0:57881] timeout=300.0>) 2022-02-02 10:16:39,805 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x3dcc3d type=client open_count=1 channel=[id: 0xe6dfc837, 0.0.0.0/0.0.0.0:39414] timeout=300.0>) 2022-02-02 10:16:41,818 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x3dcc3e type=client open_count=1 channel=[id: 0xabb5d9cb, 0.0.0.0/0.0.0.0:36498] timeout=300.0>) 2022-02-02 10:16:43,828 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x3dcc3f type=client open_count=1 channel=[id: 0x8d47ee9d, 0.0.0.0/0.0.0.0:45705] timeout=300.0>) 2022-02-02 10:16:45,838 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x3dcc40 type=client open_count=1 channel=[id: 0x7b6d44e8, 0.0.0.0/0.0.0.0:38210] timeout=300.0>) 2022-02-02 10:16:47,849 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x3dcc41 type=client open_count=1 channel=[id: 0x9fe7607d, 0.0.0.0/0.0.0.0:35556] timeout=300.0>) 2022-02-02 10:16:52,601 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x3dcc42 type=client open_count=1 channel=[id: 0x51c4503e, 0.0.0.0/0.0.0.0:53871] timeout=300.0>) 2022-02-02 10:16:54,611 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x3dcc43 type=client open_count=1 channel=[id: 0x3604c219, 0.0.0.0/0.0.0.0:36548] timeout=300.0>) 2022-02-02 10:16:56,621 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x3dcc44 type=client open_count=1 channel=[id: 0x3163feed, 0.0.0.0/0.0.0.0:48517] timeout=300.0>) 2022-02-02 10:16:58,634 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x3dcc45 type=client open_count=1 channel=[id: 0xe52e67d7, 0.0.0.0/0.0.0.0:51501] timeout=300.0>) 2022-02-02 10:17:04,075 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x3dcc46 type=client open_count=1 channel=[id: 0xc84d0baa, 0.0.0.0/0.0.0.0:60440] timeout=300.0>) 2022-02-02 10:17:06,085 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x3dcc47 type=client open_count=1 channel=[id: 0xf941ef9d, 0.0.0.0/0.0.0.0:51292] timeout=300.0>) 2022-02-02 10:17:08,099 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x3dcc48 type=client open_count=1 channel=[id: 0x9aeb3854, 0.0.0.0/0.0.0.0:37021] timeout=300.0>) 2022-02-02 10:17:10,111 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 10:17:24,980 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 10:17:37,471 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 10:17:37,473 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:17:37,489 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:17:50,286 | test | INFO | pool-8-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.27803533392 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 10:17:50,302 | test | INFO | pool-8-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 10:18:05,335 | test | INFO | pool-8-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 10:18:05,346 | test | INFO | pool-8-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0609998703 sec 2022-02-02 10:18:05,457 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.563486100676 | 11.45 GiB | 10.67 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 1.39062891506 | 11.45 GiB | 10.69 GiB | 24.25 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.38854140605 | 11.45 GiB | 10.69 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.16290726817 | 11.45 GiB | 10.73 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.70704154638 | 11.45 GiB | 10.60 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 10:18:05,457 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:18:05,457 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:18:05,693 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 10:18:06,119 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 10:18:06,427 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 86.18 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 10:18:06,427 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 10:18:16,842 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 10:18:16,848 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 10:50:54,688 | test | INFO | pool-8-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.....*................**......*****X | 150.000 | | 29.100 | ....*X | 29.100 | | 0.000 | ****************************************************************************************.......***...*****......*..*X | 100.000 | | 90.300 | ...............*****X | 90.300 | | 9.700 | *X | 9.700 | | 0.000 | *********************************************************************************************************************************.......***..**************.......**..******.......**....*****X | 159.700 | | 79.100 | .............*X | 79.100 | | 40.300 | ......*****X | 40.300 | | 0.000 | ************************************************************************************************************************************.......***..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ********.******...............*********.......**....*****X | 151.900 | | 71.300 | ...........*X | 71.300 | | 48.100 | .......*****X | 48.100 | | 7.800 | *X | 7.800 | | 0.000 | *******************************************************************************************************************************************.....................**.........******...............*X | 249.600 | | 231.000 | .........................................*****X | 231.000 | | 69.800 | ...........**X | 69.800 | | 58.700 | .........*****........********X | 101.000 | | 60.700 | ..........*X | 60.700 | | 40.300 | ......*****X | 40.300 | | 0.000 | ****************************************************************************************************************************************************************************.............................***.......*****X | 200.000 | | 79.100 | .............*X | 79.100 | | 56.600 | .........******......................**....******........**.*****X | 250.000 | | 88.800 | ...............**X | 88.800 | | 40.300 | ......****X | 40.300 | | 0.000 | ************************************************************************************************************************************************.******...............**..*****X | 100.000 | | 59.700 | .........********............................................***......****X | 330.600 | | 88.800 | ...............**X | 88.800 | | 76.800 | ............*****..............................**X | 238.000 | | 223.200 | .......................................****X | 223.200 | | 21.700 | ..**X | 21.700 | | 0.000 | ********************************************************************************************************************************.......***..*****.....*..**..*****.......*.......**....*****X | 200.000 | | 38.800 | ......*X | 38.800 | | 26.800 | ...******....................................................**..*****X | 323.200 | | 41.100 | ......*X | 41.100 | | 0.000 | *************************************************************************************************************************.......***..*****.......***..*****......................***.....*****X | 250.000 | | 48.500 | .......*X | 48.500 | | 0.000 | *************************************************************************************************************************..**..******..*X | 35.800 | | 26.100 | ...******..........***....*****..................................................................***................*****X | 550.000 | | 106.700 | ..................*X | 106.700 | | 0.000 | ***********************************************************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 10:50:55,707 | test | INFO | pool-8-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 161.200 | ............................**.......*****X | 200.000 | | 38.800 | ......**X | 38.800 | | 0.000 | ******************************************************************************************.******..............................**.....*****X | 200.000 | | 79.100 | .............**X | 79.100 | | 40.300 | ......****X | 40.300 | | 0.000 | ***********************************************************************************************************************************....................................***...........******......................**.*****X | 390.300 | | 67.900 | ...........*X | 67.900 | | 0.000 | ***************************************************************************************************************************************************.......**..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ******..............***........******...............*X | 204.100 | | 176.500 | ...............................*****X | 176.500 | | 55.600 | .........*X | 55.600 | | 50.000 | ........******.......**..*****X | 100.000 | | 19.400 | ..**X | 19.400 | | 0.000 | **********************************************************************************************************************************.......**..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ******....................................***.........********..*******X | 259.700 | | 240.300 | ..........................................*****X | 240.300 | | 38.800 | ......*X | 38.800 | | 0.000 | ******.......***..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *****************************************************************************************************************************************************************..............**....*****X | 100.000 | | 19.400 | ..*X | 19.400 | | 9.700 | ******.....................................***.......*****X | 250.000 | | 129.100 | ......................*X | 129.100 | | 119.400 | ....................******........*X | 159.700 | | 120.900 | ....................*****X | 120.900 | | 0.000 | *********************************************************************************************************************************************.......**...*******X | 59.700 | | 50.000 | ........***************.............................**.....****X | 240.300 | | 38.800 | ......**X | 38.800 | | 19.400 | ..*****.................................................................................***................****X | 550.000 | | 227.600 | ........................................**X | 227.600 | | 150.000 | ..........................****X | 150.000 | | 109.700 | ..................**X | 109.700 | | 80.600 | .............*****X | 80.600 | | 0.000 | ******************************************************************************************************************.......***..*************..............***....*****X | 150.000 | | 29.100 | ....*X | 29.100 | | 0.000 | ******..............**....*****X | 100.000 | | 19.400 | ..*X | 19.400 | | 0.000 | *********************************************************************************************************************************.......***..*****.......***...*****X | 109.700 | | 69.400 | ...........*X | 69.400 | | 40.300 | ......*****X | 40.300 | | 0.000 | *******************************************************************************************************************.......**..******.......**..*****X | 100.000 | | 19.400 | ..*X | 19.400 | | 0.000 | ******.......***..****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ************************************************************************X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 10:50:56,239 | test | INFO | pool-8-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****....................................................................**......................*****X | 39389.400 | | 25766.300 | ..........................................................*X | 25766.300 | | 20013.200 | ............................................******...........***.*****X | 25205.300 | | 16538.900 | ....................................*X | 16538.900 | | 15144.200 | .................................******...**...*****X | 17522.200 | | 15602.900 | ..................................*X | 15602.900 | | 13691.700 | ..............................******.......**......******.........................**......******.....................**....*****X | 43567.300 | | 37347.700 | ....................................................................................*X | 37347.700 | | 36262.800 | ..................................................................................*****X | 36262.800 | | 34762.300 | ..............................................................................*X | 34762.300 | | 33319.100 | ...........................................................................*****X | 33319.100 | | 28661.900 | ................................................................*X | 28661.900 | | 27565.000 | ..............................................................*****X | 27565.000 | | 14164.600 | ...............................**X | 14164.600 | | 11423.400 | .........................*****.***.******..**.******.**.*****X | 14600.100 | | 13661.300 | ..............................*X | 13661.300 | | 12774.300 | ............................*****X | 12774.300 | | 12060.200 | ..........................***..******........**..******..*X | 17913.200 | | 17617.400 | .......................................******..**.*****X | 19115.600 | | 17679.900 | .......................................**X | 17679.900 | | 17359.300 | ......................................*****..........**........******..................................*X | 39970.400 | | 39187.900 | ........................................................................................*****X | 39187.900 | | 20797.900 | ..............................................*X | 20797.900 | | 18367.200 | .........................................*****X | 18367.200 | | 14064.800 | ...............................*X | 14064.800 | | 13465.000 | .............................******...............................***........****X | 30090.700 | | 22381.400 | ..................................................**X | 22381.400 | | 18545.900 | .........................................*****X | 18545.900 | | 11672.400 | .........................*X | 11672.400 | | 11257.000 | ........................*****X | 11257.000 | | 9588.400 | .....................*X | 9588.400 | | 9492.800 | ....................******.............**...******..**X | 17121.800 | | 17119.100 | ......................................*****.**X | 17586.800 | | 17158.100 | ......................................*****X | 17158.100 | | 12965.600 | ............................*X | 12965.600 | | 11919.200 | ..........................********..******.......********.......***...*****.........***...******...........*X | 30236.700 | | 30037.700 | ...................................................................*****X | 30037.700 | | 17352.100 | ......................................*X | 17352.100 | | 14629.000 | ................................*****X | 14629.000 | | 11651.300 | .........................*X | 11651.300 | | 11446.500 | .........................******.....***.*****..***.*******X | 15653.600 | | 15124.700 | .................................*****X | 15124.700 | | 11726.200 | .........................*X | 11726.200 | | 11374.300 | .........................******......***.****X | 14605.500 | | 12175.600 | ..........................**X | 12175.600 | | 11612.400 | .........................*****.......***...****X | 15713.200 | | 15546.800 | ..................................X | 15546.800 | | 15474.900 | ..................................*X | 15474.900 | | 15153.000 | .................................*****X | 15153.000 | | 13676.600 | ..............................*X | 13676.600 | | 13357.300 | .............................*****X | 13357.300 | | 12131.800 | ..........................*X | 12131.800 | | 11919.100 | ..........................******..***.****X | 13476.700 | | 12801.100 | ............................**X | 12801.100 | | 12352.300 | ...........................******..........**..*****X | 17783.700 | | 15648.900 | ..................................*X | 15648.900 | | 14940.000 | .................................*****X | 14940.000 | | 13315.900 | .............................*X | 13315.900 | | 13052.300 | ............................******............**.....*****X | 20101.900 | | 17915.000 | ........................................*X | 17915.000 | | 16775.500 | .....................................*****X | 16775.500 | | 15492.300 | ..................................**X | 15492.300 | | 15399.500 | ..................................*****..**X | 16452.900 | | 16295.800 | ....................................*****X | 16295.800 | | 14245.500 | ...............................*X | 14245.500 | | 13939.200 | ..............................*****X | 13939.200 | | 12639.600 | ............................*X | 12639.600 | | 12473.100 | ...........................*****X | 12473.100 | | 11755.700 | .........................*X | 11755.700 | | 11619.500 | .........................******..********X | 12519.800 | | 12515.800 | ...........................*X | 12515.800 | | 12133.100 | ..........................*****X | 12133.100 | | 11514.900 | .........................**.******.......**..******..*X | 16599.200 | | 15995.500 | ...................................******X | 15995.500 | | 15393.400 | ..................................**.*****......***.*****...**X | 20081.100 | | 19673.200 | ............................................*****X | 19673.200 | | 18051.000 | ........................................*******X | 18220.300 | | 15048.400 | .................................*X | 15048.400 | | 14355.300 | ...............................******.....*******X | 16515.600 | | 14717.200 | ................................*******X | 14736.700 | | 12861.700 | ............................**X | 12861.700 | | 12224.700 | ...........................*****.....***..*****.....***.*****X | 17900.100 | | 16896.200 | .....................................*X | 16896.200 | | 16757.700 | .....................................*****X | 16757.700 | | 16308.200 | ....................................*X | 16308.200 | | 16038.300 | ...................................*****X | 16038.300 | | 13657.200 | ..............................*X | 13657.200 | | 12340.700 | ...........................*****X | 12340.700 | | 6288.000 | .............**X | 6288.000 | | 5221.300 | ..........*****........***....******........**.*****X | 14341.200 | | 13865.900 | ..............................**.*****X | 14054.100 | | 11790.700 | ..........................**X | 11790.700 | | 10991.700 | ........................*****..........***..*****X | 16285.400 | | 11824.300 | ..........................*X | 11824.300 | | 10992.600 | ........................******...**..******......**.********X | 16365.300 | | 15981.400 | ...................................*****X | 15981.400 | | 13264.400 | .............................*X | 13264.400 | | 13020.100 | ............................******..*******X | 13922.700 | | 13690.500 | ..............................*X | 13690.500 | | 13672.300 | ..............................*************X | 13758.200 | | 13015.800 | ............................**X | 13015.800 | | 12279.000 | ...........................*****X | 12279.000 | | 9172.700 | ....................*X | 9172.700 | | 8907.600 | ...................******.......**...*****.......**.*****X | 16664.300 | | 14422.600 | ................................**X | 14422.600 | | 13858.300 | ..............................*****X | 13858.300 | | 11128.900 | ........................*X | 11128.900 | | 10780.800 | .......................******.............**...*****X | 17810.200 | | 14240.600 | ...............................*X | 14240.600 | | 13552.400 | ..............................******.....**.*****X | 16403.800 | | 14239.400 | ...............................*X | 14239.400 | | 13173.500 | .............................******..***..*****....***..*****.**X | 17935.000 | | 17257.500 | ......................................*****X | 17257.500 | | 13582.300 | ..............................*X | 13582.300 | | 12756.600 | ............................*****X | 12756.600 | | 10281.200 | ......................**.******.............**.*****X | 16722.900 | | 10152.700 | ......................**X | 10152.700 | | 9128.600 | ...................*****........***....******...........*******X | 19099.200 | | 16819.200 | .....................................*X | 16819.200 | | 16784.100 | .....................................******......**.*****X | 19618.300 | | 16447.000 | ....................................**X | 16447.000 | | 15765.500 | ...................................****X | 15765.500 | | 14587.500 | ................................**X | 14587.500 | | 14323.100 | ...............................*****X | 14323.100 | | 13114.700 | .............................*X | 13114.700 | | 12919.100 | ............................*****X | 12919.100 | | 11677.300 | .........................********..........**.*****X | 16372.300 | | 12713.400 | ............................*X | 12713.400 | | 12101.100 | ..........................******........**..*****X | 16142.200 | | 13190.900 | .............................**X | 13190.900 | | 12745.000 | ............................*****X | 12745.000 | | 10897.500 | ........................*X | 10897.500 | | 9757.100 | .....................******......**...****X | 13597.300 | | 8952.100 | ...................**X | 8952.100 | | 6863.900 | ..............*****............***......*****.............***..****X | 21157.900 | | 16160.900 | ....................................**X | 16160.900 | | 14660.300 | ................................*****X | 14660.300 | | 14438.800 | ................................*X | 14438.800 | | 14119.300 | ...............................******...**...*******X | 16780.800 | | 16269.800 | ....................................*****X | 16269.800 | | 13932.400 | ..............................*X | 13932.400 | | 13902.400 | ..............................******.............**...*****X | 20760.300 | | 15348.000 | ..................................**X | 15348.000 | | 13694.500 | ..............................*****X | 13694.500 | | 11935.700 | ..........................*X | 11935.700 | | 11717.500 | .........................******..........**....******..*X | 18491.400 | | 17812.300 | .......................................*****X | 17812.300 | | 17123.600 | ......................................**X | 17123.600 | | 16896.100 | .....................................*****..***.*****X | 18101.700 | | 15948.000 | ...................................*X | 15948.000 | | 15761.500 | ...................................******.*X | 16309.100 | | 16170.900 | ....................................******....*******X | 18074.600 | | 17565.400 | .......................................*******X | 17604.000 | | 17383.400 | ......................................***.*******X | 17831.500 | | 17528.500 | .......................................******.**...******......*X | 22175.600 | | 21722.800 | ................................................*****X | 21722.800 | | 17867.300 | ........................................*X | 17867.300 | | 16865.200 | .....................................******.**.******...**.******...........*******X | 24358.700 | | 15564.100 | ..................................**X | 15564.100 | | 14018.400 | ...............................******......********.**..*****X | 18004.000 | | 17043.200 | ......................................*X | 17043.200 | | 16375.600 | ....................................******.....***.*****X | 19043.200 | | 18471.100 | .........................................*X | 18471.100 | | 18408.800 | .........................................*****X | 18408.800 | | 17508.500 | .......................................*X | 17508.500 | | 17216.100 | ......................................*******X | 17398.400 | | 17240.600 | ......................................*****X | 17240.600 | | 15379.900 | ..................................**X | 15379.900 | | 15019.200 | .................................*****X | 15041.700 | | 14989.500 | .................................**.******.*X | 15750.100 | | 15635.600 | ..................................******............**..*****X | 21483.700 | | 19222.300 | ...........................................**X | 19222.300 | | 18501.500 | .........................................****X | 18501.500 | | 16215.900 | ....................................**X | 16215.900 | | 16215.700 | ....................................*******X | 16460.000 | | 16257.100 | ....................................******......**..*****X | 19990.300 | | 17258.100 | ......................................*X | 17258.100 | | 16313.900 | ....................................******..**.*****X | 17622.300 | | 17252.700 | ......................................**X | 17252.700 | | 17035.800 | ......................................*****X | 17035.800 | | 15032.000 | .................................********........**.*****X | 18963.000 | | 17584.500 | .......................................*X | 17584.500 | | 16915.200 | .....................................*****X | 16915.200 | | 16070.800 | ...................................***.*****......***..****X | 19882.500 | | 17146.800 | ......................................**X | 17146.800 | | 16283.900 | ....................................******.....**.*****X | 18999.400 | | 17472.600 | .......................................*X | 17472.600 | | 16908.500 | .....................................*****X | 16908.500 | | 15391.200 | ..................................***.*****..**X | 16835.100 | | 16575.900 | .....................................******......**...*****X | 20514.800 | | 20331.800 | .............................................*X | 20331.800 | | 20150.000 | .............................................*****X | 20150.000 | | 16330.700 | ....................................*X | 16330.700 | | 15279.800 | ..................................******.........**..*****X | 20422.400 | | 19303.500 | ...........................................**X | 19303.500 | | 18890.400 | ..........................................****X | 18890.400 | | 18169.700 | ........................................**X | 18169.700 | | 17838.600 | .......................................******.**..******..........**.*****X | 23704.300 | | 21364.400 | ................................................*X | 21364.400 | | 20966.800 | ...............................................*****X | 20966.800 | | 17978.600 | ........................................**X | 17978.600 | | 17309.900 | ......................................****X | 17309.900 | | 16700.800 | .....................................***.*****..........***.*****X | 21950.300 | | 18961.700 | ..........................................*X | 18961.700 | | 18725.400 | .........................................*****X | 18725.400 | | 15996.800 | ...................................*X | 15996.800 | +-----------+-----------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 10:50:57,503 | test | INFO | pool-8-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 13806.500 | ...................................................................................................X | 13806.500 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 10:50:57,609 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 41.7039213182 | 11.45 GiB | 8.51 GiB | 52.50 MiB / 3.50 GiB | 6524399 / 13051600 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 33.614411652 | 11.45 GiB | 8.70 GiB | 24.25 MiB / 3.50 GiB | 6537104 / 13084611 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 44.5970225873 | 11.45 GiB | 8.34 GiB | 28.14 MiB / 3.50 GiB | 6490608 / 12992634 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 40.7216494845 | 11.45 GiB | 8.74 GiB | 0.0 Byte / 3.50 GiB | 6547824 / 13119077 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 27.0985051744 | 11.45 GiB | 8.61 GiB | 0.0 Byte / 3.50 GiB | 6509752 / 12968481 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 10:51:05,661 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 10:51:19,842 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 10:51:19,842 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 10:51:30,499 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 10:51:30,499 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 10:51:41,188 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 10:51:41,190 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 10:51:59,305 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 247.31 MiB | 326.54 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 513.57 MiB | 585.62 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.77 GiB | 50.44 GiB | 20.5688042978 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 10:51:59,305 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #5 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 10:52:03,601 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during hard_failover_recovery 2022-02-02 10:52:03,601 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : hard_failover_recovery 2022-02-02 10:52:03,601 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:724] failing over nodes [ip:172.23.105.36 port:8091 ssh_username:root, ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 10:52:12,729 | test | INFO | pool-8-thread-4 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03200006485 seconds to complete 2022-02-02 10:52:12,732 | test | INFO | pool-8-thread-4 | [common_lib:sleep:22] Sleep 8.03200006485 seconds. Reason: Wait after rebalance complete 2022-02-02 10:57:29,880 | test | INFO | pool-8-thread-25 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03200006485 seconds to complete 2022-02-02 10:57:29,881 | test | INFO | pool-8-thread-25 | [common_lib:sleep:22] Sleep 8.03200006485 seconds. Reason: Wait after rebalance complete 2022-02-02 11:02:37,915 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 11:02:37,917 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 11:02:37,924 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 11:14:47,790 | test | INFO | pool-8-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.............................................................................*******X | 106.000 | | 24.000 | ................*******X | 24.000 | | 0.000 | ********.................*********...........................................................*******X | 104.000 | | 90.000 | .................................................................*******X | 90.000 | | 40.000 | ............................********......********X | 48.000 | | 32.000 | ......................********......********X | 40.000 | | 16.000 | ..........********..................********...............................................*******X | 104.000 | | 8.000 | ....*********..........................................********.........................*******X | 98.000 | | 32.000 | ......................********...........................*******X | 68.000 | | 44.000 | ...............................*******X | 44.000 | | 16.000 | ..........********..................*****************............********.........................*******X | 90.000 | | 80.000 | .........................................................*******X | 80.000 | | 48.000 | ..................................*******X | 48.000 | | 40.000 | ............................********.....................................................*******X | 112.000 | | 10.000 | ......*********.....................................................*******X | 82.000 | | 38.000 | ..........................*******X | 38.000 | | 8.000 | ....********...............................................................................................*******X | 136.000 | | 44.000 | ...............................*******X | 44.000 | | 10.000 | ......*********.................................................................*******X | 98.000 | | 70.000 | ..................................................********......********....................*******X | 106.000 | | 28.000 | ...................********X | 28.000 | | 26.000 | ..................*******X | 26.000 | | 18.000 | ............********....................*******X | 46.000 | | 0.000 | *********.............*******X | 18.000 | | 16.000 | ..........********.........................................*******X | 72.000 | | 20.000 | .............********..................................................********X | 88.000 | | 42.000 | .............................********.....................*******X | 69.700 | | 28.300 | ...................********...........................................*******X | 86.000 | | 30.000 | .....................*******X | 30.000 | | 28.000 | ...................*********......................................*******X | 80.000 | | 38.000 | ..........................*******X | 38.000 | | 26.000 | ..................********.............................*******X | 66.000 | | 32.000 | ......................*********.............*******X | 50.000 | | 0.000 | ********................................................********X | 66.000 | | 36.000 | .........................*******X | 36.000 | | 0.000 | ********...................*******X | 26.000 | | 8.000 | ....********.............*..............................*******X | 66.000 | | 34.000 | ........................*******X | 34.000 | | 16.000 | ..........********..................*******X | 40.000 | | 7.400 | ....*********.**X | 8.600 | +--------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 11:14:47,818 | test | INFO | pool-8-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................................................................................................*******X | 376.000 | | 40.000 | .........*******X | 40.000 | | 0.000 | ********...........*********.......................*******X | 130.000 | | 74.000 | ..................*******X | 74.000 | | 32.000 | .......********.....*******X | 50.000 | | 24.000 | .....****************X | 24.000 | | 16.000 | ...*********.......********................*******X | 104.000 | | 18.000 | ...********...............********X | 74.000 | | 72.000 | ..................*******X | 72.000 | | 50.000 | ............********..************X | 58.000 | | 58.000 | ..............**X | 58.000 | | 46.000 | ...........***X | 46.000 | | 46.000 | ...........***X | 46.000 | | 34.000 | ........*X | 34.000 | | 34.000 | ........**X | 34.000 | | 34.000 | ........****....**X | 50.000 | | 50.000 | ............****X | 50.000 | | 32.000 | .......********.............*******X | 82.000 | | 74.000 | ..................*******X | 74.000 | | 26.000 | .....********............****************X | 70.000 | | 28.000 | ......********............*******X | 72.000 | | 44.000 | ..........*******X | 44.000 | | 26.000 | .....*********..........................*******X | 122.000 | | 24.000 | .....*******X | 24.000 | | 10.000 | .********.................................*******X | 132.000 | | 46.000 | ...........********X | 46.000 | | 42.000 | ..........****************..................*******X | 112.000 | | 32.000 | .......********.........********X | 64.000 | | 50.000 | ............*******X | 50.000 | | 24.000 | .....*******X | 24.000 | | 8.000 | .********..********.................*******X | 82.000 | | 46.000 | ...........********X | 46.000 | | 34.000 | ........*******X | 34.000 | | 26.000 | .....********.......********X | 50.000 | | 0.000 | ********..............*******X | 54.000 | | 16.000 | ...********.....********..********X | 44.000 | | 16.000 | ...***************X | 18.000 | | 16.000 | ...********..*********..............*******X | 76.000 | | 8.000 | .********....*******X | 24.000 | | 8.000 | .*********....*******X | 26.000 | | 18.000 | ...*******X | 18.000 | | 0.000 | ********.............********X | 50.000 | | 16.000 | ...********************************X | 16.000 | | 0.000 | **X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 11:14:48,375 | test | INFO | pool-8-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...********........................********.......********...........*********.................*******X | 37982.800 | | 34638.600 | .......................................................********.....*******X | 37736.500 | | 34606.400 | .......................................................*******X | 34606.400 | | 22262.100 | ...................................********X | 22262.100 | | 15841.600 | ........................********................*******X | 25483.200 | | 25466.500 | ........................................********.............*********..............*******X | 41775.200 | | 40097.500 | ................................................................********....................*******X | 52504.600 | | 42369.000 | ....................................................................*******X | 42369.000 | | 33543.000 | .....................................................********X | 33543.000 | | 25204.600 | ........................................********........*******X | 30049.000 | | 28686.800 | .............................................*******X | 28686.800 | | 26409.900 | ..........................................*********............*******..............................*...............X | 61155.900 | | 61155.900 | ..................................................................................................*.X | 61155.900 | | 61155.900 | ..................................................................................................***.X | 61155.900 | | 40367.600 | .................................................................********.........*X | 46378.900 | | 46378.900 | ..........................................................................**X | 46378.900 | | 46378.900 | ..........................................................................*X | 46378.900 | | 46378.900 | ..........................................................................*X | 46378.900 | | 32230.100 | ...................................................*******X | 32230.100 | | 31955.400 | ...................................................********......********......********X | 39698.000 | | 32826.200 | ....................................................********...........*******X | 39234.400 | | 32356.800 | ...................................................********.....********...********...*******X | 38861.300 | | 29384.000 | ...............................................********.........................********X | 45197.800 | | 42234.000 | ....................................................................*******X | 42234.000 | | 28943.800 | ..............................................********...........********...*******X | 37432.200 | | 29367.600 | ...............................................********...********X | 31448.600 | | 24782.700 | .......................................*******X | 24782.700 | | 21588.700 | ..................................********.........................********.********X | 37726.400 | | 20592.300 | ................................********........*******X | 25391.700 | | 21403.700 | .................................********..........********.....................*******X | 39883.600 | | 32775.100 | ....................................................*******X | 32775.100 | | 27104.700 | ...........................................********.*******X | 27920.300 | | 26872.300 | ..........................................X | 26872.300 | | 24610.300 | .......................................********..********..........*******X | 32284.500 | | 27999.600 | ............................................*******X | 27999.600 | | 23171.600 | ....................................********...........********X | 29723.800 | | 24092.000 | ......................................********.*******X | 24617.200 | | 24551.700 | .......................................*******X | 24551.700 | | 20577.900 | ................................********X | 20577.900 | | 19778.100 | ...............................********................*******X | 29847.100 | | 22600.000 | ...................................*******X | 22600.000 | | 20364.400 | ................................*********........*******X | 25651.600 | | 19057.000 | ..............................********.....***X | 22232.800 | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 11:14:49,654 | test | INFO | pool-8-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------+-------+ | 8.600 | ...................................................................................................X | 8.600 | +-------+------------------------------------------------------------------------------------------------------+-------+ 2022-02-02 11:14:50,851 | test | INFO | pool-8-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------+-----------+ | 22232.800 | ...................................................................................................*X | 22232.800 | +-----------+-------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 11:14:50,947 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 11:14:50,947 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 11:14:50,967 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 11:14:50,967 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 11:15:00,808 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 11:15:00,815 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 11:15:00,918 | test | INFO | pool-8-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 0.650650650651 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 25.8166624462 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 31.7091739627 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.363135487102 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 27.3453346856 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 11:15:00,936 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 11:15:10,960 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 11:15:21,000 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 17.0 2022-02-02 11:15:31,056 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 41.26 2022-02-02 11:15:41,098 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 51.27 2022-02-02 11:15:51,138 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 60.86 2022-02-02 11:16:01,184 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 66.67 2022-02-02 11:16:11,220 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 67.08 2022-02-02 11:16:21,256 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 67.92 2022-02-02 11:16:31,292 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 68.8 2022-02-02 11:16:41,331 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 69.59 2022-02-02 11:16:51,377 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 70.57 2022-02-02 11:17:01,420 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 71.59 2022-02-02 11:17:11,480 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 72.38 2022-02-02 11:17:21,516 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 73.63 2022-02-02 11:17:31,563 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 74.71 2022-02-02 11:17:41,598 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 75.54 2022-02-02 11:17:51,650 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 76.62 2022-02-02 11:18:01,687 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 77.29 2022-02-02 11:18:11,729 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 78.26 2022-02-02 11:18:21,763 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 79.16 2022-02-02 11:18:31,805 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 79.97 2022-02-02 11:18:41,839 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 80.72 2022-02-02 11:18:51,882 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 81.45 2022-02-02 11:19:01,931 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 82.31 2022-02-02 11:19:11,967 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 83.18 2022-02-02 11:19:22,005 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 84.04 2022-02-02 11:19:32,046 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 84.93 2022-02-02 11:19:42,092 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 85.63 2022-02-02 11:19:52,127 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 86.26 2022-02-02 11:20:02,163 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 86.81 2022-02-02 11:20:12,194 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 87.38 2022-02-02 11:20:22,236 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 88.15 2022-02-02 11:20:32,282 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 88.69 2022-02-02 11:20:42,319 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 89.15 2022-02-02 11:20:52,371 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 89.71 2022-02-02 11:21:02,415 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 90.18 2022-02-02 11:21:12,460 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 90.59 2022-02-02 11:21:22,497 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 91.01 2022-02-02 11:21:32,533 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 91.66 2022-02-02 11:21:42,581 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 91.9 2022-02-02 11:21:52,619 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 92.27 2022-02-02 11:22:02,667 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 92.64 2022-02-02 11:22:12,720 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 92.97 2022-02-02 11:22:22,763 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 93.38 2022-02-02 11:22:32,805 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 93.9 2022-02-02 11:22:42,848 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 94.29 2022-02-02 11:22:52,892 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 94.46 2022-02-02 11:23:02,928 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 94.92 2022-02-02 11:23:12,964 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 95.25 2022-02-02 11:23:23,007 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 95.42 2022-02-02 11:23:33,062 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 95.85 2022-02-02 11:23:43,105 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.04 2022-02-02 11:23:53,144 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.38 2022-02-02 11:24:03,180 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.64 2022-02-02 11:24:13,230 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.88 2022-02-02 11:24:23,272 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 97.15 2022-02-02 11:24:33,303 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 97.41 2022-02-02 11:24:43,338 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 97.69 2022-02-02 11:24:53,371 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.01 2022-02-02 11:25:03,405 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.2 2022-02-02 11:25:13,448 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.39 2022-02-02 11:25:23,483 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.61 2022-02-02 11:25:33,517 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.84 2022-02-02 11:25:43,556 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.97 2022-02-02 11:25:53,588 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.17 2022-02-02 11:26:03,621 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.42 2022-02-02 11:26:13,653 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.53 2022-02-02 11:26:23,687 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.67 2022-02-02 11:26:33,721 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.78 2022-02-02 11:26:43,753 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.81 2022-02-02 11:26:53,785 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: running, progress: 99.94 2022-02-02 11:27:08,836 | test | INFO | pool-8-thread-17 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 11:27:08,854 | test | INFO | pool-8-thread-17 | [task:check:468] Rebalance completed with progress: 100% in 727.936000109 sec 2022-02-02 11:28:41,388 | test | INFO | pool-8-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******....................................................................................................*******X | 161.600 | | 7.600 | ...*******X | 7.600 | | 0.000 | *********..........................................................................................**.........*****X | 160.200 | | 25.800 | ..............*X | 25.800 | | 19.500 | ...........******..............***..*****X | 46.700 | | 33.800 | ...................*X | 33.800 | | 25.300 | ..............******...........................**..........*****X | 85.600 | | 25.100 | ..............*X | 25.100 | | 9.300 | ....******..................................**.............***X | 84.800 | | 84.800 | ...................................................*X | 84.800 | | 71.500 | ...........................................**X | 71.500 | | 54.000 | ................................****X | 54.000 | | 18.600 | ..........**X | 18.600 | | 13.500 | .......*****..................***......*****...........***.....*****..X | 81.300 | | 81.300 | .................................................*X | 81.300 | | 79.500 | ................................................*****X | 79.500 | | 35.200 | ....................*X | 35.200 | | 25.500 | ..............******.............*X | 46.200 | | 45.200 | ..........................******..........**......*****X | 70.400 | | 27.600 | ................*X | 27.600 | | 16.900 | .........******.......................**....X | 61.100 | | 61.100 | ....................................**X | 61.100 | | 61.100 | ....................................*X | 61.100 | | 12.600 | ......*X | 12.600 | | 0.000 | ******................***........*****...**X | 43.900 | | 34.900 | ....................*****...........***......****X | 62.800 | | 31.800 | ..................**X | 31.800 | | 26.200 | ...............*****...................***.****X | 59.400 | | 43.200 | .........................***..*****...................***.*****X | 78.500 | | 41.700 | ........................*X | 41.700 | | 34.400 | ....................*****X | 34.400 | | 19.600 | ...........*X | 19.600 | | 15.200 | ........*****X | 15.200 | | 13.000 | .......X | 13.000 | | 9.200 | ....*.******.........................**...*****X | 55.600 | | 27.700 | ................**..******..............*X | 54.500 | | 53.000 | ...............................******...........................**...*****X | 100.300 | | 44.400 | ..........................*X | 44.400 | | 31.700 | ..................******...**..*****X | 40.000 | | 20.700 | ...........*X | 20.700 | | 17.900 | ..........******.........................................***.......****X | 96.800 | | 24.400 | ..............**X | 24.400 | | 12.000 | ......*****..........................***...****X | 58.800 | | 30.800 | ..................**X | 30.800 | | 28.600 | ................****X | 28.600 | | 25.400 | ..............**X | 25.400 | | 21.000 | ...........*****..........***...*****X | 42.000 | | 37.700 | ......................**........******.............*X | 71.500 | | 48.600 | .............................*****X | 48.600 | | 14.800 | ........**....******........*X | 34.600 | | 34.200 | ....................*****X | 34.200 | | 19.400 | ...........*X | 19.400 | | 17.200 | .........*****X | 17.200 | | 15.700 | ........*X | 15.700 | | 15.200 | ........******.....................***.........****X | 63.200 | | 31.000 | ..................**X | 31.000 | | 15.000 | ........*****X | 15.000 | | 7.600 | ...**...******...........................**.*****X | 56.600 | | 8.000 | ...*X | 8.000 | | 0.000 | ******...................**.........*****X | 46.400 | | 29.000 | ................*X | 29.000 | | 16.600 | .........*****X | 16.600 | | 13.400 | .......*******X | 13.400 | | 9.200 | ....*********.......................**.......*****X | 57.200 | | 19.200 | ..........*X | 19.200 | | 16.800 | .........******....................*X | 49.200 | | 40.400 | ........................*****X | 40.400 | | 2.200 | ********.............***....*****...**X | 34.600 | | 28.800 | ................****X | 28.800 | | 18.800 | ..........**X | 18.800 | | 17.000 | .........****X | 17.000 | | 14.700 | ........X | 14.700 | | 8.000 | ...**.******...............................**.......*****X | 70.600 | | 53.200 | ...............................*X | 53.200 | | 49.600 | .............................*****X | 49.600 | | 19.000 | ..........*X | 19.000 | | 9.600 | ....******..........................**.......*****X | 62.800 | | 41.100 | ........................*X | 41.100 | | 34.400 | ....................*****X | 34.400 | | 19.600 | ...........*X | 19.600 | | 18.800 | ..........******....**X | 24.600 | | 23.200 | .............*****........**X | 36.400 | | 36.000 | .....................*****X | 36.000 | | 28.600 | ................**..******............**..*****X | 53.400 | | 11.200 | .....*X | 11.200 | | 0.000 | ******.......***....*****................................***.......*****X | 81.600 | | 36.200 | .....................*X | 36.200 | | 19.200 | ..........***X | 19.200 | +--------+----------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 11:28:41,642 | test | INFO | pool-8-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******..........................................................................*X | 91.200 | | 91.200 | .........................................................................*****X | 91.200 | | 32.400 | .........................********X | 32.600 | | 6.100 | ...**..................******..............................................................................**X | 122.400 | | 108.700 | .......................................................................................****X | 108.700 | | 8.500 | .....**X | 8.500 | | 6.800 | ....******.....................................................**....*****X | 76.800 | | 17.800 | .............*X | 17.800 | | 9.300 | ......******.......................................**........*****X | 66.700 | | 13.600 | ..........**X | 13.600 | | 0.000 | ******...........................................**.................*****X | 73.500 | | 63.200 | ..................................................*X | 63.200 | | 54.000 | ...........................................******.**.*X | 56.700 | | 56.700 | .............................................*X | 56.700 | | 56.700 | .............................................**.................**..*****X | 80.200 | | 33.000 | .........................*X | 33.000 | | 26.200 | ....................******......................................**..........*****X | 85.300 | | 63.200 | ..................................................*X | 63.200 | | 51.800 | .........................................******.***...*****............***....****X | 76.000 | | 48.000 | ......................................**X | 48.000 | | 40.500 | ................................****X | 40.500 | | 36.100 | ............................**X | 36.100 | | 32.200 | .........................*****X | 32.200 | | 24.800 | ...................**.******.............................**............*****X | 76.500 | | 69.600 | .......................................................*X | 69.600 | | 55.000 | ...........................................**X | 55.000 | | 55.000 | ...........................................**X | 55.000 | | 31.800 | ........................**.*****X | 32.400 | | 26.600 | ....................**..******...................................**....*****X | 76.600 | | 45.400 | ....................................*X | 45.400 | | 41.700 | .................................******......*X | 49.300 | | 47.900 | ......................................*****X | 47.900 | | 27.000 | .....................*X | 27.000 | | 26.300 | ....................******..........**X | 38.200 | | 34.500 | ...........................****X | 34.500 | | 5.100 | ...**X | 5.100 | | 2.200 | *****................................***.......*****.......***.......****X | 66.500 | | 63.400 | ..................................................**X | 63.400 | | 60.400 | ................................................*****...........**X | 73.900 | | 68.100 | ......................................................****X | 68.100 | | 25.300 | ...................**X | 25.300 | | 16.500 | ............****X | 16.500 | | 15.000 | ...........***...*****............***..*****.....................................***........*****X | 91.700 | | 25.400 | ...................*X | 25.400 | | 11.300 | ........******.....................**....******................**...........******.............*X | 91.500 | | 87.800 | ......................................................................*****X | 87.800 | | 58.000 | ..............................................*X | 58.000 | | 47.700 | .....................................*****X | 47.700 | | 25.600 | ...................**.....******.............*X | 47.400 | | 35.700 | ............................*****X | 35.700 | | 28.200 | ......................**.....*****X | 34.400 | | 22.800 | .................********.........................................***......****X | 80.000 | | 74.000 | ...........................................................X | 74.000 | | 51.000 | ........................................*X | 51.000 | | 46.000 | ....................................*****X | 46.000 | | 27.900 | .....................*X | 27.900 | | 22.800 | .................*****X | 22.800 | | 12.800 | .........*X | 12.800 | | 11.400 | ........*****X | 11.400 | | 9.800 | .......**...******.....................*X | 40.200 | | 35.900 | ............................*****X | 35.900 | | 16.800 | ............*X | 16.800 | | 15.400 | ...........*****X | 15.400 | | 8.000 | .....**X | 8.000 | | 7.600 | .....**************.********....**.******..................................**...........*****X | 70.400 | | 37.100 | .............................**X | 37.100 | | 32.600 | .........................*****..........**X | 44.200 | | 36.200 | ............................****X | 36.200 | | 4.000 | ..**X | 4.000 | | 1.800 | ******......................**........******.........*X | 50.000 | | 45.600 | ....................................*****X | 45.600 | | 13.400 | .........*X | 13.400 | | 5.800 | ...******..**.******...............................**.............*****X | 62.800 | | 40.600 | ................................**X | 40.600 | | 30.300 | .......................*****X | 30.300 | | 24.600 | ...................*X | 24.600 | | 19.300 | ..............*****X | 19.300 | | 19.200 | ..............**........******............*X | 43.400 | | 38.000 | ..............................*****X | 38.000 | | 29.000 | ......................**..........******.....................*X | 66.400 | | 49.800 | .......................................*****X | 49.800 | | 1.800 | *X | 1.800 | | 0.000 | ******....***...*****.....**X | 15.200 | | 13.400 | .........*****X | 13.400 | | 1.800 | *X | 1.800 | | 0.000 | ******..............**.............******........................................................***.****X | 103.000 | | 22.800 | .................**X | 22.800 | | 5.800 | ...***X | 5.800 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 11:28:42,473 | test | INFO | pool-8-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.......................................................*******X | 25552.100 | | 22295.900 | ...............................................*******X | 22295.900 | | 19407.000 | ........................................********.................********X | 27223.900 | | 26962.800 | .........................................................*******X | 26962.800 | | 17048.900 | ...................................********........*********....................**.**X | 30489.100 | | 30489.100 | ................................................................*X | 30489.100 | | 30489.100 | ................................................................X | 30489.100 | | 29725.700 | ...............................................................*******X | 30017.200 | | 23386.400 | .................................................********..............***.*****.............***..****X | 37204.800 | | 36287.400 | .............................................................................***.*****..................**.*****X | 45687.000 | | 35843.100 | ............................................................................*X | 35843.100 | | 35823.500 | ............................................................................*****X | 35823.500 | | 34184.400 | ........................................................................*X | 34184.400 | | 34105.800 | ........................................................................******..**...******...*X | 38034.900 | | 37923.300 | ................................................................................*****X | 37923.300 | | 34205.700 | ........................................................................**X | 34205.700 | | 33384.000 | ......................................................................*****......***..****X | 36864.700 | | 29332.100 | ..............................................................**X | 29332.100 | | 28252.900 | ...........................................................****X | 28252.900 | | 25833.400 | ......................................................**X | 25833.400 | | 25703.800 | ......................................................*****................***..*****...........*******X | 39403.800 | | 34496.400 | .........................................................................X | 34496.400 | | 34496.400 | .........................................................................X | 34496.400 | | 34199.800 | ........................................................................******........................**...*****X | 46408.200 | | 37898.800 | ................................................................................*X | 37898.800 | | 37171.700 | ...............................................................................*****X | 37171.700 | | 33025.500 | ......................................................................*X | 33025.500 | | 31535.500 | ..................................................................*****X | 31535.500 | | 28332.200 | ............................................................*******X | 28543.200 | | 28184.100 | ...........................................................**.*****X | 28649.400 | | 27874.500 | ...........................................................*X | 27874.500 | | 27317.300 | .........................................................******..............**..*****X | 34752.000 | | 32677.500 | .....................................................................**.*****.**X | 33513.500 | | 33055.300 | ......................................................................****X | 33055.300 | | 31073.300 | .................................................................**X | 31073.300 | | 30714.500 | .................................................................****X | 30714.500 | | 29385.700 | ..............................................................***.****X | 29980.000 | | 26888.900 | ........................................................**X | 26888.900 | | 26708.500 | ........................................................*****.....................***..****X | 37350.300 | | 29870.900 | ...............................................................**X | 29870.900 | | 29826.500 | ...............................................................****X | 29826.500 | | 27919.500 | ...........................................................**X | 27919.500 | | 27013.500 | .........................................................*****...**.******....**.*****X | 31273.400 | | 31242.100 | ..................................................................********.*X | 31918.300 | | 31599.500 | ...................................................................*****X | 31599.500 | | 26884.400 | ........................................................*X | 26884.400 | | 26434.800 | .......................................................*****X | 26434.800 | | 22849.600 | ................................................*X | 22849.600 | | 22643.100 | ...............................................******.......**.*****X | 26436.400 | | 22010.500 | ..............................................*X | 22010.500 | | 21152.000 | ............................................*****X | 21152.000 | | 19401.500 | ........................................*X | 19401.500 | | 19094.300 | ........................................******........**.*****X | 23661.300 | | 20987.600 | ............................................********...*X | 22444.400 | | 21515.400 | .............................................********...******........*X | 26553.900 | | 26378.800 | .......................................................*****X | 26378.800 | | 19945.100 | .........................................*X | 19945.100 | | 18145.300 | ......................................******....**...*****X | 21369.700 | | 16600.400 | ..................................*X | 16600.400 | | 15841.600 | .................................******.********.....***.****X | 19155.100 | | 18344.100 | ......................................**X | 18344.100 | | 18269.500 | ......................................****X | 18269.500 | | 18059.800 | .....................................**X | 18059.800 | | 17893.500 | .....................................*****....***..****X | 20553.700 | | 20518.600 | ...........................................**X | 20518.600 | | 19960.700 | ..........................................****X | 19960.700 | | 17729.600 | .....................................**X | 17729.600 | | 17668.100 | .....................................*****................***...****X | 26724.900 | | 21342.000 | ............................................**X | 21342.000 | | 20412.500 | ..........................................*****..**X | 20908.600 | | 20693.100 | ...........................................*****...*.**..******....*X | 25488.300 | | 25355.900 | .....................................................*****X | 25355.900 | | 17823.800 | .....................................*X | 17823.800 | | 16437.200 | ..................................******............**..*****X | 23149.900 | | 20395.900 | ..........................................*X | 20395.900 | | 19847.200 | .........................................*****X | 19847.200 | | 17152.900 | ...................................*X | 17152.900 | | 16278.500 | ..................................******.**.******..........**..*****X | 22817.800 | | 18278.300 | ......................................*X | 18278.300 | | 16883.000 | ...................................******.....***..****X | 20109.000 | | 16117.600 | .................................**X | 16117.600 | | 15204.900 | ...............................******.....**.******.**.******........**...****X | 23699.100 | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 11:28:43,767 | test | INFO | pool-8-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 19.200 | ...................................................................................................X | 19.200 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 11:28:44,944 | test | INFO | pool-8-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------+-------+ | 5.800 | ...................................................................................................X | 5.800 | | 0.000 | X | 0.000 | +-------+------------------------------------------------------------------------------------------------------+-------+ 2022-02-02 11:28:44,980 | test | INFO | pool-8-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 23699.100 | ...................................................................................................X | 23699.100 | | 14487.900 | ............................................................X | 14487.900 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 11:28:45,095 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 11:29:00,397 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 11:29:14,473 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 11:29:14,473 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 11:29:26,006 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 11:29:26,006 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 11:29:36,661 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 11:29:36,661 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 11:29:54,720 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 46000 | 9.77 GiB | 236.64 MiB | 340.57 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 4.88 GiB | 508.36 MiB | 528.29 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 30021250 | 2.50 GiB | 1.67 GiB | 52.73 GiB | 24.6241545572 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 11:29:55,210 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 11:29:55,210 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 11:30:01,703 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 11:30:01,703 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 11:30:08,280 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 29% 2022-02-02 11:30:08,280 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 1% 2022-02-02 11:31:02,727 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 11:31:03,171 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 11:31:03,173 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 11:31:33,311 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 11:31:33,706 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 11:31:33,707 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 11:32:03,878 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 11:32:04,283 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 11:32:04,285 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 11:35:05,638 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'73c2983641d002ee053dcc827c74c721', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=dad1761325209efb5538b443dd6b236d', u'status': u'notRunning'} - rebalance failed 2022-02-02 11:35:05,661 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 11:35:05,661 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643830504486L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:35:04.486Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 484814768d5e99a935654d71251df1e5'} 2022-02-02 11:35:05,661 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643830504485L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:35:04.485Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 11:35:05,661 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.177', u'tstamp': 1643830502751L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:35:02.751Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 11:35:05,663 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643830324481L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:32:04.481Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 484814768d5e99a935654d71251df1e5"} 2022-02-02 11:35:05,663 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643830324473L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T11:32:04.473Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 11:35:05,663 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643830276318L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:31:16.318Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 11:35:05,663 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643830265605L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:31:05.605Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 11:35:05,663 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643830258398L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:30:58.398Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 11:35:05,663 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643830257788L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:30:57.788Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 11:35:05,664 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643830232819L, u'shortText': u'message', u'serverTime': u'2022-02-02T11:30:32.819Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup 2022-02-02 11:35:05,664 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'73c2983641d002ee053dcc827c74c721', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=dad1761325209efb5538b443dd6b236d', u'status': u'notRunning'} - rebalance failed self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'73c2983641d002ee053dcc827c74c721', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=dad1761325209efb5538b443dd6b236d', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 4895.707s OK During the test, Remote Connections: 256, Disconnections: 256 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_5 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=hard_failover_and_delta_recovery_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_6', 'nodes_init': '5', 'GROUP': 'hard_failover_and_delta_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 6, 'randomize_value': 'True'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 11:35:11,375 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 11:35:11,750 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 11:37:20,730 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.740999937 seconds to complete 2022-02-02 11:37:20,739 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 11:37:25,819 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x4f4340 type=client open_count=1 channel=[id: 0x9db86faa, 0.0.0.0/0.0.0.0:48518] timeout=300.0>) 2022-02-02 11:37:29,707 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x4f4341 type=client open_count=1 channel=[id: 0x4fc89462, 0.0.0.0/0.0.0.0:38690] timeout=300.0>) 2022-02-02 11:37:31,717 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x4f4342 type=client open_count=1 channel=[id: 0x14509e4e, 0.0.0.0/0.0.0.0:55962] timeout=300.0>) 2022-02-02 11:37:33,729 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x4f4343 type=client open_count=1 channel=[id: 0x46552e70, 0.0.0.0/0.0.0.0:54175] timeout=300.0>) 2022-02-02 11:37:35,739 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x4f4344 type=client open_count=1 channel=[id: 0x1ddaea90, 0.0.0.0/0.0.0.0:50337] timeout=300.0>) 2022-02-02 11:37:41,454 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x4f4345 type=client open_count=1 channel=[id: 0xa5be66f0, 0.0.0.0/0.0.0.0:51812] timeout=300.0>) 2022-02-02 11:37:43,466 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x4f4346 type=client open_count=1 channel=[id: 0x8d8ce3d7, 0.0.0.0/0.0.0.0:51267] timeout=300.0>) 2022-02-02 11:37:45,476 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x4f4347 type=client open_count=1 channel=[id: 0xd488b7b7, 0.0.0.0/0.0.0.0:36492] timeout=300.0>) 2022-02-02 11:37:47,486 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x4f4348 type=client open_count=1 channel=[id: 0x2d414310, 0.0.0.0/0.0.0.0:50742] timeout=300.0>) 2022-02-02 11:37:49,500 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x4f4349 type=client open_count=1 channel=[id: 0xf0d050e5, 0.0.0.0/0.0.0.0:53341] timeout=300.0>) 2022-02-02 11:37:54,846 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x4f434a type=client open_count=1 channel=[id: 0x500f83ba, 0.0.0.0/0.0.0.0:60608] timeout=300.0>) 2022-02-02 11:37:56,857 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x4f434b type=client open_count=1 channel=[id: 0xaaf20714, 0.0.0.0/0.0.0.0:37190] timeout=300.0>) 2022-02-02 11:37:58,868 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x4f434c type=client open_count=1 channel=[id: 0x1a40919b, 0.0.0.0/0.0.0.0:57283] timeout=300.0>) 2022-02-02 11:38:00,878 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x4f434d type=client open_count=1 channel=[id: 0xab326857, 0.0.0.0/0.0.0.0:50092] timeout=300.0>) 2022-02-02 11:38:06,184 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x4f434e type=client open_count=1 channel=[id: 0x2a486907, 0.0.0.0/0.0.0.0:52609] timeout=300.0>) 2022-02-02 11:38:08,196 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x4f434f type=client open_count=1 channel=[id: 0x1c269e7f, 0.0.0.0/0.0.0.0:41759] timeout=300.0>) 2022-02-02 11:38:10,206 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x4f4350 type=client open_count=1 channel=[id: 0xa3d303d7, 0.0.0.0/0.0.0.0:59556] timeout=300.0>) 2022-02-02 11:38:12,217 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x4f4351 type=client open_count=1 channel=[id: 0xed331028, 0.0.0.0/0.0.0.0:39846] timeout=300.0>) 2022-02-02 11:38:17,632 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x4f4352 type=client open_count=1 channel=[id: 0x0a766086, 0.0.0.0/0.0.0.0:37333] timeout=300.0>) 2022-02-02 11:38:19,644 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x4f4353 type=client open_count=1 channel=[id: 0x14a934f0, 0.0.0.0/0.0.0.0:46114] timeout=300.0>) 2022-02-02 11:38:21,654 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x4f4354 type=client open_count=1 channel=[id: 0x7f3dd89d, 0.0.0.0/0.0.0.0:45255] timeout=300.0>) 2022-02-02 11:38:23,664 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x4f4355 type=client open_count=1 channel=[id: 0x8af0affd, 0.0.0.0/0.0.0.0:53479] timeout=300.0>) 2022-02-02 11:38:25,676 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x4f4356 type=client open_count=1 channel=[id: 0xa9dd0553, 0.0.0.0/0.0.0.0:45796] timeout=300.0>) 2022-02-02 11:38:31,344 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x4f4357 type=client open_count=1 channel=[id: 0xb341f5b6, 0.0.0.0/0.0.0.0:55151] timeout=300.0>) 2022-02-02 11:38:33,355 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x4f4358 type=client open_count=1 channel=[id: 0xbc07caca, 0.0.0.0/0.0.0.0:55396] timeout=300.0>) 2022-02-02 11:38:35,365 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x4f4359 type=client open_count=1 channel=[id: 0x4c8068ea, 0.0.0.0/0.0.0.0:40895] timeout=300.0>) 2022-02-02 11:38:37,377 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x4f435a type=client open_count=1 channel=[id: 0xb7bff358, 0.0.0.0/0.0.0.0:48872] timeout=300.0>) 2022-02-02 11:38:42,661 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x4f435b type=client open_count=1 channel=[id: 0xc35adfc1, 0.0.0.0/0.0.0.0:48472] timeout=300.0>) 2022-02-02 11:38:44,673 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x4f435c type=client open_count=1 channel=[id: 0xfe514f91, 0.0.0.0/0.0.0.0:40864] timeout=300.0>) 2022-02-02 11:38:46,683 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x4f435d type=client open_count=1 channel=[id: 0x45b09aa1, 0.0.0.0/0.0.0.0:34761] timeout=300.0>) 2022-02-02 11:38:48,694 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 11:39:03,525 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 11:39:15,954 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 11:39:15,954 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 11:39:15,971 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 11:39:29,819 | test | INFO | pool-9-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.57756354075 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 11:39:29,835 | test | INFO | pool-9-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 11:39:44,917 | test | INFO | pool-9-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 11:39:44,930 | test | INFO | pool-9-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.1119999886 sec 2022-02-02 11:39:45,038 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.513013013013 | 11.45 GiB | 10.67 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.713927855711 | 11.45 GiB | 10.69 GiB | 24.25 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 0.225592179471 | 11.45 GiB | 10.67 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.162805259862 | 11.45 GiB | 10.74 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.43983973958 | 11.45 GiB | 10.57 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 11:39:45,039 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 11:39:45,039 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 11:39:45,200 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 11:39:45,628 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 11:39:45,944 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 84.33 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 11:39:45,944 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 11:39:54,923 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 11:39:54,930 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 12:08:36,257 | test | INFO | pool-9-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ******................................................................................**....................*****X | 1350.000 | | 270.000 | ...................**X | 270.000 | | 0.000 | **************************************......................................**..........*****X | 650.000 | | 130.000 | ........*X | 130.000 | | 0.000 | *************************************...........***....*****....**X | 268.800 | | 238.800 | ................*****X | 238.800 | | 70.000 | ....*X | 70.000 | | 50.000 | ..******...***.****X | 100.000 | | 20.000 | **X | 20.000 | | 0.000 | **********************************************************************..**.******.........***.....*******X | 230.000 | | 170.000 | ...........*****X | 170.000 | | 90.000 | .....**.*****X | 100.000 | | 20.000 | *X | 20.000 | | 0.000 | *************************************************************************************************************************...********......**.*****X | 140.000 | | 20.000 | *X | 20.000 | | 0.000 | **********************************************************************************************************************************************************************************..***.*****X | 50.000 | | 10.000 | *X | 10.000 | | 0.000 | ******..**..******......***.*******X | 150.000 | | 140.000 | .........*****X | 140.000 | | 20.000 | *X | 20.000 | | 0.000 | ******.....**..*****X | 100.000 | | 20.000 | *X | 20.000 | | 0.000 | *************************************************************************************************************************..**.*****X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | ********.******.....................**..*****X | 330.000 | | 50.000 | ..*X | 50.000 | | 0.000 | ********************************************************************************************************************************************************************..******...............*X | 230.000 | | 220.000 | ...............*****X | 220.000 | | 140.000 | .........**.*****X | 150.000 | | 30.000 | .**X | 30.000 | | 0.000 | *****.....***..*****X | 100.000 | | 20.000 | *X | 20.000 | | 0.000 | ********************************************************************************************************************************.....***...*******X | 110.000 | | 90.000 | .....*****X | 90.000 | | 10.000 | *X | 10.000 | | 0.000 | ******************************************************************************************************************************************************************..**.*****X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | *********************************************************************************************************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 12:08:36,453 | test | INFO | pool-9-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.........................................................**....................******.......*X | 62297.200 | | 57109.700 | ............................................................................*****X | 57109.700 | | 52570.400 | ......................................................................***....*****............**X | 64900.000 | | 64391.700 | ......................................................................................****X | 64391.700 | | 39675.600 | ....................................................**X | 39675.600 | | 31923.900 | ..........................................******..**......*****.............***..****X | 48771.300 | | 35241.200 | ..............................................**X | 35241.200 | | 30262.900 | ........................................*****........................**...........******......................**..****X | 73778.500 | | 63789.800 | .....................................................................................**X | 63789.800 | | 60489.700 | ................................................................................****X | 60489.700 | | 57587.100 | .............................................................................**.*****X | 58672.400 | | 46686.500 | ..............................................................*X | 46686.500 | | 39696.000 | ....................................................*****X | 39696.000 | | 17034.800 | ......................*X | 17034.800 | | 15573.300 | ....................********X | 16213.500 | | 13762.300 | .................*****X | 13762.300 | | 6966.200 | ........********..............**.....******............**.*****X | 30945.200 | | 30343.000 | ........................................*X | 30343.000 | | 30290.700 | ........................................*****X | 30290.700 | | 26552.100 | ..................................**X | 26552.100 | | 25688.400 | .................................*****....***.****X | 29300.600 | | 26403.900 | ..................................**X | 26403.900 | | 25699.400 | .................................*****.********............**....*****X | 37674.600 | | 37475.400 | .................................................*X | 37475.400 | | 36036.600 | ...............................................*****X | 36036.600 | | 26262.500 | ..................................*X | 26262.500 | | 23597.800 | ..............................*****X | 23597.800 | | 20997.900 | ...........................**.*****X | 21408.500 | | 15519.600 | ....................**X | 15519.600 | | 12841.600 | ................*****.***.*****X | 14705.700 | | 13449.500 | .................*******X | 13514.300 | | 12977.800 | ................*X | 12977.800 | | 12153.000 | ...............******..**.*****X | 14382.300 | | 12485.200 | ...............***.*****.*.*X | 14076.200 | | 13525.500 | .................******..********..**.******.........**.....******.**X | 28205.900 | | 26199.000 | ..................................****X | 26199.000 | | 20090.600 | ..........................**X | 20090.600 | | 19274.500 | .........................*****X | 19274.500 | | 16604.100 | .....................*X | 16604.100 | | 15423.000 | ...................******...........**......******......*X | 31895.400 | | 30734.000 | ........................................*****X | 30734.000 | | 23670.900 | ...............................*X | 23670.900 | | 22118.400 | ............................*****X | 22118.400 | | 15027.200 | ...................**X | 15027.200 | | 13598.200 | .................*****....***..*****X | 18259.900 | | 15934.600 | ....................*X | 15934.600 | | 14310.300 | ..................******........**...*****X | 22376.900 | | 15536.900 | ....................*X | 15536.900 | | 14006.400 | .................******..*X | 14934.000 | | 13847.700 | .................*****X | 13847.700 | | 13047.900 | ................***..****X | 14189.500 | | 10406.200 | .............**X | 10406.200 | | 8594.500 | ..........******.....**..******..**.******...*X | 17924.500 | | 16335.200 | .....................*****X | 16335.200 | | 13796.400 | .................***..*****..********X | 16854.000 | | 10608.400 | .............*X | 10608.400 | | 9217.300 | ...........******...........**...******.**.*****X | 21375.400 | | 21272.000 | ...........................**X | 21272.000 | | 19144.300 | ........................****X | 19144.300 | | 12317.600 | ...............**X | 12317.600 | | 11576.200 | ..............******......**.*****X | 16382.100 | | 12240.700 | ...............*X | 12240.700 | | 11646.100 | ..............*****X | 11646.100 | | 11360.900 | ..............********.......********X | 16551.100 | | 10349.300 | .............*X | 10349.300 | | 9517.000 | ...........******....*******X | 12514.500 | | 9492.300 | ...........**.******.............**X | 19280.800 | | 19110.600 | ........................****X | 19110.600 | | 12291.500 | ...............X | 12291.500 | | 10598.700 | .............*X | 10598.700 | | 10311.800 | ............******....*X | 13116.300 | | 12593.800 | ................*****X | 12593.800 | | 9657.500 | ............********...............***..****X | 22835.600 | | 13030.900 | ................**X | 13030.900 | | 10215.800 | ............******.**...******...................**.*****X | 27386.500 | | 16541.200 | .....................*X | 16541.200 | | 15559.700 | ....................******.........**..*****X | 23947.900 | | 19242.100 | .........................**X | 19242.100 | | 18612.500 | ........................****X | 18612.500 | | 16716.900 | .....................**X | 16716.900 | | 15036.100 | ...................*****X | 15036.100 | | 13859.700 | .................**..*******X | 15224.000 | | 14211.300 | ..................*****X | 14211.300 | | 13232.400 | ................**.*****X | 13748.000 | | 10788.600 | .............**X | 10788.600 | | 10204.200 | ............*****............***...*****X | 21293.600 | | 19667.700 | .........................*X | 19667.700 | | 18586.900 | ........................*****X | 18586.900 | | 16727.600 | .....................*******X | 16945.300 | | 15712.100 | ....................*X | 15712.100 | | 14271.800 | ..................*****X | 14271.800 | | 10048.900 | ............**X | 10048.900 | | 9277.800 | ...........*****...***..******........*******X | 19063.500 | | 12647.100 | ................*X | 12647.100 | | 10362.300 | .............******...**.....******...........**X | 25054.700 | | 24901.200 | ................................*****X | 24901.200 | | 12168.800 | ...............*X | 12168.800 | | 9713.600 | ............******.....**..*****X | 15442.700 | | 13356.700 | .................*X | 13356.700 | | 10953.100 | .............******...**...*****X | 15199.100 | | 12826.800 | ................**X | 12826.800 | | 10924.400 | .............******..**...******........*X | 19937.300 | | 19664.000 | .........................*****X | 19664.000 | | 15028.800 | ...................*X | 15028.800 | | 14926.800 | ...................******....**.******..**X | 20559.500 | | 19303.200 | .........................****X | 19303.200 | | 11891.100 | ...............**X | 11891.100 | | 11442.900 | ..............******..*X | 13042.300 | | 12376.100 | ...............*****X | 12376.100 | | 6415.500 | .......*X | 6415.500 | | 5384.800 | ......******................**......******......**X | 26027.400 | | 24752.000 | ................................****X | 24752.000 | | 15352.100 | ...................**X | 15352.100 | | 13984.500 | .................*****.....***.******..*******X | 19457.300 | | 15831.000 | ....................*X | 15831.000 | | 15290.100 | ...................******......**.*****X | 20077.200 | | 12490.000 | ...............**X | 12490.000 | | 10929.100 | .............*****...***.*****............***...*****X | 24444.400 | | 21505.100 | ............................*X | 21505.100 | | 20191.800 | ..........................*****X | 20191.800 | | 11378.300 | ..............*X | 11378.300 | | 10271.900 | ............******........**...******...**X | 20629.400 | | 19818.600 | .........................****X | 19818.600 | | 13911.700 | .................**X | 13911.700 | | 13488.200 | .................******......**..*****X | 19193.100 | | 13152.900 | ................*X | 13152.900 | | 10810.700 | .............******..**..******.........***..****X | 21724.800 | | 19674.000 | .........................********.*...*******X | 22249.200 | | 16731.000 | .....................*X | 16731.000 | | 15604.700 | ....................********.******...*X | 18731.000 | | 18699.000 | ........................*****X | 18699.000 | | 15532.400 | ....................**X | 15532.400 | | 14517.400 | ..................*****..***.*****X | 16962.800 | | 15267.300 | ...................*X | 15267.300 | | 13100.600 | ................*****X | 13100.600 | | 10108.700 | ............**..******.................***..****X | 25452.400 | | 20159.700 | ..........................**X | 20159.700 | | 19622.100 | .........................*****X | 19622.100 | | 17911.700 | .......................*X | 17911.700 | | 17264.400 | ......................*****X | 17264.400 | | 15963.100 | ....................*X | 15963.100 | | 15705.300 | ....................*****X | 15705.300 | | 14651.400 | ..................********...*******X | 16909.200 | | 14742.600 | ..................**X | 14742.600 | | 13143.700 | ................*****X | 13143.700 | | 7812.100 | .........*X | 7812.100 | | 6747.100 | ........******............**.....******.......*******X | 24823.400 | | 14684.000 | ..................**X | 14684.000 | | 11709.300 | ..............*****....***....******.......*******X | 22819.800 | | 12395.000 | ...............*X | 12395.000 | | 8382.600 | ..........******.***....*****..........********X | 19688.100 | | 12545.600 | ................*X | 12545.600 | | 10471.500 | .............*****X | 10471.500 | | 9540.500 | ...........**..******...............**.*****X | 22427.600 | | 14707.700 | ..................**X | 14707.700 | | 13597.000 | .................*****..***..******.....*******X | 20219.300 | | 13729.100 | .................*X | 13729.100 | | 11852.600 | ...............******....**.*****X | 16065.600 | | 9859.100 | ............**X | 9859.100 | | 9828.500 | ............*****......**X | 14633.000 | | 13836.700 | .................*****X | 13836.700 | | 6944.600 | ........**.******.........................**...*****X | 28654.300 | | 20468.100 | ..........................*X | 20468.100 | | 18310.700 | .......................*****X | 18310.700 | | 13628.100 | .................**X | 13628.100 | | 13317.600 | .................******.....**.******..*X | 19660.100 | | 19556.700 | .........................*****X | 19556.700 | | 16752.600 | .....................**.******.*X | 17989.500 | | 16984.800 | ......................*****X | 16984.800 | | 12652.800 | ................**X | 12652.800 | | 11662.900 | ..............*****X | 11662.900 | | 5710.600 | ......*X | 5710.600 | | 4594.300 | .....******..............*X | 15452.500 | +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 12:08:37,538 | test | INFO | pool-9-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 560.000 | ................................................................**.................*****X | 700.000 | | 140.000 | ...............**X | 140.000 | | 0.000 | **************************************................................................................................***....................****X | 850.000 | | 170.000 | ...................**X | 170.000 | | 0.000 | *************************************....**....******...................*******X | 230.000 | | 30.000 | ..**X | 30.000 | | 0.000 | ******..............**...*****X | 150.000 | | 30.000 | ..*X | 30.000 | | 0.000 | ***********************************************************************..................**.....******........................**.......*****X | 460.000 | | 140.000 | ...............**X | 140.000 | | 40.000 | ...********.*****X | 50.000 | | 10.000 | *X | 10.000 | | 0.000 | *******************************************************************************************************************.******....*X | 50.000 | | 40.000 | ...******..........**...*****X | 150.000 | | 30.000 | ..*X | 30.000 | | 0.000 | ******....***.****X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | **************************************************************************************************************************************************************************....**.******.....**..********X | 110.000 | | 100.000 | ..........****X | 100.000 | | 60.000 | ......**X | 60.000 | | 40.000 | ...*****X | 40.000 | | 0.000 | *********************************************************************************************************************************************.******....**X | 50.000 | | 40.000 | ...*****X | 40.000 | | 0.000 | **.******...................**..*****X | 190.000 | | 30.000 | ..*X | 30.000 | | 0.000 | *********************************************************************************************************************************************************..*.......**..*****X | 100.000 | | 20.000 | .**...******........................*X | 249.600 | | 210.400 | .......................*****X | 210.400 | | 50.400 | ....*X | 50.400 | | 40.000 | ...*****X | 40.000 | | 0.000 | ********..............***...*****X | 150.000 | | 30.000 | ..*X | 30.000 | | 0.000 | ************************************************************************************************************************....***...*****....*********.................................**.......*****X | 440.000 | | 200.000 | ......................*X | 200.000 | | 150.000 | ................*****X | 150.000 | | 70.000 | .......*X | 70.000 | | 50.000 | ....*****X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | *********************************************************************************************************************************....**.*************X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | ******************************************************************************************************************X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 12:08:38,792 | test | INFO | pool-9-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 18701.400 | ...................................................................................................X | 18701.400 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 12:08:38,905 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 38.7096774194 | 11.45 GiB | 8.48 GiB | 52.50 MiB / 3.50 GiB | 6483597 / 13035353 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 41.3616585616 | 11.45 GiB | 8.70 GiB | 24.25 MiB / 3.50 GiB | 6525888 / 13014326 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 48.0670762929 | 11.45 GiB | 8.49 GiB | 28.14 MiB / 3.50 GiB | 6539814 / 12995520 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 37.9047864792 | 11.45 GiB | 8.47 GiB | 0.0 Byte / 3.50 GiB | 6502385 / 13060699 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 43.509798931 | 11.45 GiB | 8.80 GiB | 0.0 Byte / 3.50 GiB | 6455116 / 12909502 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 12:08:48,029 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 12:09:02,007 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 12:09:02,007 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 12:09:12,713 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 12:09:12,713 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 12:09:23,562 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 12:09:23,562 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 12:09:41,773 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 253.61 MiB | 350.74 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 512.57 MiB | 502.50 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.71 GiB | 52.81 GiB | 18.7715395242 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 12:09:41,773 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #6 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 12:09:46,072 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during hard_failover_recovery 2022-02-02 12:09:46,073 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : hard_failover_recovery 2022-02-02 12:09:46,073 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:724] failing over nodes [ip:172.23.105.36 port:8091 ssh_username:root, ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 12:09:55,201 | test | INFO | pool-9-thread-8 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03299999237 seconds to complete 2022-02-02 12:09:55,204 | test | INFO | pool-9-thread-8 | [common_lib:sleep:22] Sleep 8.03299999237 seconds. Reason: Wait after rebalance complete 2022-02-02 12:15:12,368 | test | INFO | pool-9-thread-5 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03499984741 seconds to complete 2022-02-02 12:15:12,371 | test | INFO | pool-9-thread-5 | [common_lib:sleep:22] Sleep 8.03499984741 seconds. Reason: Wait after rebalance complete 2022-02-02 12:20:20,407 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 12:20:20,408 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 12:20:20,415 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 12:33:17,076 | test | INFO | pool-9-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******....................................................................................................*******X | 146.000 | | 56.000 | .....................................********...........................********X | 96.000 | | 24.000 | ...............********......................................................................*******X | 126.000 | | 24.000 | ...............*********................*******X | 48.000 | | 32.000 | ....................********......********.............................................*******X | 106.000 | | 82.000 | .......................................................*******X | 82.000 | | 48.000 | ...............................********....................*******X | 76.000 | | 32.000 | ....................*****************..................*******X | 58.000 | | 52.000 | ..................................********X | 52.000 | | 16.000 | .........********.............................X | 58.000 | | 58.000 | ......................................******X | 58.000 | | 48.000 | ...............................********..........................*******X | 86.000 | | 84.000 | ........................................................*******X | 84.000 | | 82.000 | .......................................................*******X | 82.000 | | 50.000 | .................................********.....................................********X | 104.000 | | 70.000 | ..............................................*******X | 70.000 | | 0.000 | ********..............................................*******X | 68.000 | | 28.000 | ..................*********............................*******X | 70.000 | | 42.000 | ...........................*******X | 42.000 | | 24.000 | ...............*******X | 24.000 | | 10.000 | .....********.............********................................................********X | 98.000 | | 52.000 | ..................................*******X | 52.000 | | 50.000 | .................................*******X | 50.000 | | 26.000 | ................*******X | 26.000 | | 0.000 | ********................................********...............********X | 70.000 | | 8.000 | ....********...............********..........*******X | 44.000 | | 26.000 | ................********...********X | 30.000 | | 10.000 | .....*******X | 10.000 | | 8.000 | ....********............................................********X | 72.000 | | 68.000 | .............................................*******X | 68.000 | | 34.000 | ......................********.....*******X | 42.000 | | 26.000 | ................********...............................................................*******X | 118.000 | | 38.000 | .........................********.................********X | 64.000 | | 10.000 | .....********............................*******X | 50.000 | | 8.000 | ....********.......*******X | 18.000 | | 16.000 | .........********.........................*******X | 52.000 | | 8.000 | ....*********....................................*******X | 60.000 | | 32.000 | ....................*******X | 32.000 | | 24.000 | ...............*********.......********.......................*******X | 68.000 | | 32.000 | ....................*******X | 32.000 | | 16.000 | .........****************X | 16.000 | | 8.000 | ....*****************.....*******X | 16.000 | | 0.000 | ********..........*******X | 16.000 | | 0.000 | **X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 12:33:17,140 | test | INFO | pool-9-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******.....................................................................*******X | 44123.700 | | 32132.700 | .................................................********...*******X | 34092.300 | | 27833.000 | ...........................................********................********X | 38219.800 | | 23497.600 | ....................................*******X | 23497.600 | | 20636.200 | ...............................********.....********....********.......................********.................*X | 51501.600 | | 51501.600 | ................................................................................*X | 51501.600 | | 51501.600 | ................................................................................*X | 51501.600 | | 51501.600 | ................................................................................*X | 51501.600 | | 49775.600 | .............................................................................********X | 49775.600 | | 40664.200 | ...............................................................*******X | 40664.200 | | 27504.500 | ..........................................*******X | 27504.500 | | 20961.200 | ................................*******X | 20961.200 | | 20040.600 | ..............................********.........*********...........********..............*******X | 41242.000 | | 37019.200 | .........................................................********..********..............................*****X | 56920.100 | | 56920.100 | .........................................................................................**..........*****X | 63230.400 | | 63230.400 | ..................................................................................................*.X | 63230.400 | | 42406.800 | ..................................................................********......*******X | 46268.400 | | 34612.700 | .....................................................*******X | 34612.700 | | 28526.600 | ............................................****************..............********X | 37694.700 | | 27790.000 | ..........................................*******X | 27790.000 | | 26443.000 | ........................................********..............*******X | 35107.600 | | 25774.400 | .......................................********.....********........X | 34070.100 | | 34070.100 | ....................................................******X | 34070.100 | | 22187.200 | ..................................********.....********..............*******X | 34224.500 | | 29993.900 | ..............................................*******X | 29993.900 | | 25569.300 | .......................................********..................*******X | 37265.600 | | 27700.300 | ..........................................*********.********..........*******X | 34166.400 | | 31612.300 | ................................................********...*******X | 32949.700 | | 20057.100 | ..............................********X | 20057.100 | | 18878.500 | ............................********.................********...*******X | 31311.800 | | 29800.800 | ..............................................********..................*******X | 41147.300 | | 33526.900 | ....................................................*******X | 33526.900 | | 27104.100 | .........................................*******X | 27104.100 | | 24861.700 | ......................................*********.*******X | 25508.400 | | 22488.300 | ..................................********......*******X | 26408.200 | | 23752.100 | ....................................*******X | 23752.100 | | 18666.300 | ............................********..................*******X | 30257.500 | | 26699.300 | .........................................X | 26699.300 | | 24967.800 | ......................................*******X | 24967.800 | | 18971.400 | .............................********......*******X | 23023.300 | | 21626.300 | .................................********X | 21626.300 | | 16847.800 | .........................********....********...*******X | 21297.000 | | 16941.300 | .........................*********..*******X | 17742.200 | | 15243.000 | .......................********..********......********X | 20453.800 | | 19916.200 | ..............................*******X | 19916.200 | | 17448.100 | ..........................********.********..........**X | 24599.300 | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 12:33:17,371 | test | INFO | pool-9-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******...........*******X | 16.000 | | 0.000 | ********...........*******X | 16.000 | | 8.000 | ....********...*........*******X | 24.000 | | 0.000 | ********.................................*******X | 48.000 | | 24.000 | ...............********X | 24.000 | | 8.000 | ....********.....................................................................................*******X | 128.000 | | 16.000 | ..........********...............................................................*******X | 106.000 | | 40.000 | ...........................********X | 40.000 | | 24.000 | ...............********......********.*********...........................*******X | 72.000 | | 32.000 | .....................********...................................*******X | 82.000 | | 56.000 | ......................................********X | 56.000 | | 40.000 | ...........................********...................................................................*******X | 136.000 | | 100.000 | .....................................................................*******X | 100.000 | | 48.000 | ................................*******X | 48.000 | | 8.000 | ....********..........................................................*******X | 90.000 | | 0.000 | ********...........................................................********X | 84.000 | | 42.000 | ............................********.............*******X | 60.000 | | 16.000 | ..........********.................********X | 40.000 | | 18.000 | ...........********.....................................*******X | 70.000 | | 34.000 | ......................********......********....................********X | 70.000 | | 44.000 | .............................******X | 44.000 | | 8.000 | ....********...............................................................................................********X | 142.000 | | 52.000 | ...................................*******X | 52.000 | | 36.000 | ........................********..........********............................********X | 90.000 | | 30.000 | ....................*******X | 30.000 | | 10.000 | ......***************X | 10.000 | | 0.000 | *********..........................................*******X | 60.000 | | 18.000 | ...........********..........................................................*******X | 100.000 | | 76.000 | ....................................................*******X | 76.000 | | 0.000 | ********....................................................................................*******X | 120.000 | | 28.000 | ..................*****************.....................********............*******X | 74.000 | | 52.000 | ...................................*******X | 52.000 | | 26.000 | .................********..................*******X | 52.000 | | 24.000 | ...............********...........................*********............................*******X | 102.000 | | 0.000 | ********............................*********........********.........*******X | 64.000 | | 8.000 | ....*********..................................*******X | 56.000 | | 50.000 | ..................................*******X | 50.000 | | 8.000 | ....*********.................*******X | 32.000 | | 24.000 | ...............*******X | 24.000 | | 16.000 | ..........*********.......*******X | 26.000 | | 8.000 | ....**X | 8.000 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 12:33:18,648 | test | INFO | pool-9-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 24599.300 | ...................................................................................................X | 24599.300 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 12:33:19,801 | test | INFO | pool-9-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------+-------+ | 8.000 | ...................................................................................................*X | 8.000 | +-------+-------------------------------------------------------------------------------------------------------+-------+ 2022-02-02 12:33:19,897 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 12:33:19,898 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 12:33:19,917 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 12:33:19,917 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 12:33:30,897 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 12:33:30,905 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 12:33:31,150 | test | INFO | pool-9-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 0.763072304228 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 20.2883885656 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 23.8578680203 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.250438266967 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 23.5145385588 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 12:33:31,170 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 12:33:41,197 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 12:33:51,253 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 37.06 2022-02-02 12:34:01,299 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:34:11,341 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:34:21,380 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:34:31,415 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:34:41,461 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:34:51,499 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:35:01,535 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:35:11,575 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:35:21,609 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:35:31,650 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:35:41,707 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.33 2022-02-02 12:35:51,753 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 53.45 2022-02-02 12:36:01,805 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 54.69 2022-02-02 12:36:11,842 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 55.6 2022-02-02 12:36:21,878 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 58.05 2022-02-02 12:36:31,911 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 61.04 2022-02-02 12:36:41,961 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 61.93 2022-02-02 12:36:52,006 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 65.35 2022-02-02 12:37:02,053 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 65.39 2022-02-02 12:37:12,091 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 72.18 2022-02-02 12:37:22,138 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: running, progress: 77.17 2022-02-02 12:37:37,188 | test | INFO | pool-9-thread-25 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 12:37:37,207 | test | INFO | pool-9-thread-25 | [task:check:468] Rebalance completed with progress: 100% in 246.057999849 sec 2022-02-02 12:48:22,864 | test | INFO | pool-9-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******....................................................................................................*******X | 281.200 | | 40.000 | .............*********......................**..........*****X | 131.200 | | 41.200 | .............*X | 41.200 | | 15.200 | ....******.......**..*****X | 40.000 | | 9.000 | ..*X | 9.000 | | 0.000 | ******........**..******.........**.....*****X | 67.600 | | 61.700 | ....................*X | 61.700 | | 46.400 | ...............******.............**........*****X | 104.200 | | 49.700 | ................*X | 49.700 | | 37.200 | ............******............*X | 72.600 | | 67.500 | .......................*****X | 67.500 | | 45.400 | ...............**********..******.*X | 55.300 | | 53.800 | ..................******....**X | 67.100 | | 66.400 | ......................**X | 66.400 | | 66.400 | ......................*X | 66.400 | | 17.800 | .....**X | 17.800 | | 7.600 | .******...**...******..........*X | 52.000 | | 51.500 | .................*****X | 51.500 | | 33.800 | ...........*X | 33.800 | | 31.900 | ..........******.............**....*****X | 80.100 | | 47.600 | ...............*X | 47.600 | | 32.000 | ..........******....**..**X | 49.600 | | 49.600 | ................****X | 49.700 | | 45.700 | ...............*****X | 45.700 | | 39.700 | .............**.******.*X | 45.800 | | 41.800 | .............*****X | 41.800 | | 7.800 | .***.*****..............**X | 49.800 | | 48.500 | ................******........**.....*****X | 86.300 | | 61.200 | ....................*X | 61.200 | | 47.600 | ...............*****X | 47.600 | | 38.800 | ............*X | 38.800 | | 37.800 | ............*****X | 37.800 | | 21.600 | ......*X | 21.600 | | 19.200 | .....******.........**...*****X | 51.600 | | 36.800 | ............**X | 36.800 | | 32.400 | ..........*****...********X | 42.000 | | 17.100 | .....*X | 17.100 | | 16.900 | .....******........*X | 42.000 | | 36.900 | ............*****X | 36.900 | | 25.100 | .......**.******......***.****X | 47.500 | | 37.400 | ............**X | 37.400 | | 34.900 | ...........****X | 34.900 | | 33.200 | ..........**X | 33.200 | | 30.300 | .........*****...***..********.*****X | 47.700 | | 18.100 | .....*X | 18.100 | | 5.900 | .******..............**.....*****X | 59.200 | | 57.600 | ...................*X | 57.600 | | 57.200 | ...................*****X | 57.200 | | 19.000 | .....**X | 19.000 | | 9.300 | ..*****.......***..*****X | 36.300 | | 28.800 | .........*X | 28.800 | | 25.600 | ........*****X | 25.600 | | 21.400 | ......*X | 21.400 | | 21.000 | ......******..............**......*****X | 76.100 | | 73.100 | ........................**X | 73.100 | | 55.300 | ..................****X | 55.300 | | 11.300 | ...********....**X | 25.000 | | 24.600 | .......*****X | 24.600 | | 4.000 | *X | 4.000 | | 0.000 | ****************...******...............**X | 51.800 | | 48.800 | ................*****X | 48.800 | | 34.000 | ...........*X | 34.000 | | 32.600 | ..........******....**.*****X | 45.600 | | 38.200 | ............**.*****X | 40.000 | | 9.400 | ..*X | 9.400 | | 0.000 | ******....***...****X | 20.800 | | 19.200 | .....**X | 19.200 | | 13.400 | ...********..******....*X | 28.400 | | 24.400 | .......*****X | 24.400 | | 15.400 | ....***.*****X | 17.200 | | 9.800 | ..**..******......*X | 32.200 | | 25.000 | .......*****X | 25.000 | | 19.200 | .....**..****************..******......**X | 47.600 | | 43.600 | ..............****X | 43.600 | | 36.200 | ...........***.*****X | 36.600 | | 20.800 | ......*X | 20.800 | | 15.000 | ....********.*****X | 17.200 | | 9.800 | ..**X | 9.800 | | 7.600 | .*****...***.******.**.*****X | 22.800 | | 5.400 | ********.........*X | 28.600 | | 25.000 | .......******...........**......*****X | 70.400 | | 24.000 | .......**X | 24.000 | | 9.600 | ..****X | 9.600 | | 2.200 | **X | 2.200 | | 0.000 | ******......**..*****X | 22.800 | | 5.400 | *X | 5.400 | | 3.600 | ******......*******X | 21.000 | | 9.400 | ..**X | 9.400 | | 5.800 | .****X | 5.800 | +--------+--------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 12:48:23,434 | test | INFO | pool-9-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.........................********....................*********...............................**X | 30695.900 | | 30695.900 | ...........................................................................*X | 30695.900 | | 30695.900 | ...........................................................................*X | 30695.900 | | 30490.500 | ..........................................................................X | 30490.500 | | 30057.200 | .........................................................................*******X | 30057.200 | | 27502.600 | ...................................................................*******............*******X | 32327.500 | | 28600.700 | ......................................................................********...........**X | 32922.100 | | 32922.100 | .................................................................................**X | 32922.100 | | 32922.100 | .................................................................................**...***X | 34361.600 | | 34361.600 | ....................................................................................**X | 34361.600 | | 34361.600 | ....................................................................................X | 34361.600 | | 29390.600 | ........................................................................********..............********.............*******X | 40124.200 | | 36796.300 | ..........................................................................................*******X | 36796.300 | | 30991.300 | ............................................................................****X | 31219.400 | | 31219.400 | ............................................................................*X | 31219.400 | | 31219.400 | ............................................................................X | 31219.400 | | 30629.200 | ...........................................................................**..******...**.*****X | 33197.300 | | 24597.800 | ............................................................*X | 24597.800 | | 24522.700 | ............................................................******.........**.******.........................**..*****X | 39346.100 | | 30979.200 | ............................................................................*X | 30979.200 | | 30054.600 | .........................................................................**X | 30054.600 | | 30054.600 | .........................................................................***......................**..****X | 39612.500 | | 31059.300 | ............................................................................***.****X | 31656.100 | | 29892.100 | .........................................................................***.....*****.***......*****......*******X | 37140.000 | | 33721.300 | ...................................................................................**X | 33721.300 | | 32660.700 | ................................................................................****X | 32660.700 | | 28852.800 | ......................................................................**X | 28852.800 | | 27905.400 | ....................................................................******.**.******.......**...*****X | 32602.800 | | 32106.800 | ...............................................................................*X | 32106.800 | | 30166.900 | ..........................................................................*****X | 30166.900 | | 22616.300 | .......................................................*X | 22616.300 | | 22470.600 | .......................................................*****X | 22470.600 | | 20509.100 | ..................................................*X | 20509.100 | | 18933.700 | ..............................................******...........**....******...*X | 26151.800 | | 26078.000 | ...............................................................*****X | 26078.000 | | 21553.400 | ....................................................**X | 21553.400 | | 21393.200 | ....................................................*****....***.*****...*******X | 24803.700 | | 21990.700 | .....................................................**X | 21990.700 | | 21596.300 | ....................................................*****.................***......*****.......**X | 33382.500 | | 32407.600 | ...............................................................................****X | 32407.600 | | 27509.900 | ...................................................................**X | 27509.900 | | 27221.400 | ..................................................................*****......***...****X | 30567.700 | | 30531.400 | ...........................................................................**X | 30531.400 | | 29458.000 | ........................................................................*****X | 29458.000 | | 23979.900 | ..........................................................*X | 23979.900 | | 22468.900 | ......................................................*****X | 22468.900 | | 20943.700 | ...................................................**.******.*******X | 21993.500 | | 18584.300 | .............................................*X | 18584.300 | | 17424.900 | ..........................................******.....**.******.**.********.******.....**.******...**X | 24521.600 | | 24426.000 | ...........................................................****X | 24426.000 | | 20942.900 | ...................................................**X | 20942.900 | | 20572.200 | ..................................................****X | 20572.200 | | 19768.900 | ................................................**X | 19768.900 | | 19540.800 | ...............................................****X | 19540.800 | | 19034.600 | ..............................................**X | 19034.600 | | 18967.600 | ..............................................*******X | 19061.200 | | 18869.500 | ..............................................*****X | 18869.500 | | 18515.000 | .............................................**.******......**.*****X | 21669.700 | | 18309.900 | ............................................*X | 18309.900 | | 18199.900 | ............................................******.....*X | 20233.500 | | 20061.300 | ................................................******..**..*****X | 21438.200 | | 20450.400 | .................................................*X | 20450.400 | | 19956.900 | ................................................******....**.*****X | 21782.400 | | 19461.400 | ...............................................**X | 19461.400 | | 18787.800 | .............................................*****X | 18787.800 | | 17748.300 | ...........................................********..*X | 18800.000 | | 18703.200 | .............................................*****X | 18703.200 | | 18589.600 | .............................................*X | 18589.600 | | 18000.200 | ...........................................*****X | 18000.200 | | 15833.700 | ......................................**.******..........**..******.*X | 21515.300 | | 20865.800 | ...................................................*****X | 20865.800 | | 17998.600 | ...........................................*X | 17998.600 | | 17774.700 | ...........................................******.....**..******.*X | 21011.200 | | 20811.700 | ..................................................*****X | 20811.700 | | 19264.800 | ...............................................**X | 19264.800 | | 18764.600 | .............................................****X | 18764.600 | | 18001.700 | ...........................................**X | 18001.700 | | 17753.500 | ...........................................*****X | 17753.500 | | 17287.000 | ..........................................*******X | 17508.500 | | 16354.500 | .......................................*X | 16354.500 | | 16172.700 | .......................................******...********........*******X | 20806.400 | | 16233.600 | .......................................*X | 16233.600 | | 15803.400 | ......................................******.....***.*****....*******X | 19885.100 | | 15214.800 | ....................................**X | 15214.800 | | 14126.200 | ..................................*****.........***...*****..**X | 19733.600 | | 19513.300 | ...............................................*****X | 19513.300 | | 13770.000 | .................................*X | 13770.000 | | 12721.200 | ..............................******.....*X | 14538.700 | | 14303.200 | ..................................******.....**...*****X | 17487.300 | +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 12:48:23,851 | test | INFO | pool-9-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******......................................................***.****X | 54.400 | | 17.300 | ................**X | 17.300 | | 16.100 | ...............*****..............***............****X | 41.400 | | 28.800 | ............................X | 28.800 | | 23.700 | .......................*X | 23.700 | | 11.800 | ...........******...............**.............*****X | 39.300 | | 25.800 | .........................*X | 25.800 | | 17.300 | ................******..................................**.......*****X | 57.400 | | 33.800 | .................................**...*******X | 37.200 | | 25.300 | ........................******................................................**...........................*****X | 98.000 | | 62.600 | ..............................................................*X | 62.600 | | 38.800 | ......................................*****X | 38.800 | | 32.900 | ................................**....******...........................**..................*****X | 80.900 | | 72.100 | ........................................................................*X | 72.100 | | 54.000 | ......................................................*****X | 54.000 | | 36.300 | ....................................**.******..............**..*****X | 53.500 | | 41.700 | .........................................**......******...............*X | 62.100 | | 53.600 | .....................................................*****X | 53.600 | | 13.700 | ............**X | 13.700 | | 11.500 | ..........*****.........................................................***.....................****X | 87.300 | | 75.500 | ............................................................................**X | 75.500 | | 62.100 | ..............................................................X | 62.100 | | 62.100 | ..............................................................*X | 62.100 | | 62.100 | ..............................................................**.........**X | 70.900 | | 69.200 | .....................................................................****X | 69.200 | | 39.700 | .......................................**X | 39.700 | | 34.600 | ..................................*****.........................***...........****X | 70.400 | | 27.800 | ...........................**X | 27.800 | | 16.900 | ................*****...................***....*****X | 40.000 | | 26.700 | ..........................*X | 26.700 | | 22.800 | ......................******.......**....******..............................**...............**X | 78.200 | | 78.200 | ..............................................................................**X | 78.200 | | 62.000 | ..............................................................*X | 62.000 | | 38.400 | ......................................******....**.............*****X | 55.200 | | 19.900 | ...................**X | 19.900 | | 9.600 | ........*****............***......*****X | 26.600 | | 26.500 | ..........................*X | 26.500 | | 23.000 | ......................********....******...............................**......*****X | 63.100 | | 39.500 | .......................................**X | 39.500 | | 38.400 | ......................................****X | 38.400 | | 31.000 | ..............................**X | 31.000 | | 24.200 | .......................****X | 24.200 | | 22.900 | ......................X | 22.900 | | 22.700 | ......................*X | 22.700 | | 21.100 | ....................******.................................**..............*****X | 67.500 | | 44.000 | ...........................................*X | 44.000 | | 31.700 | ...............................********...*****X | 34.900 | | 34.800 | ..................................*X | 34.800 | | 32.200 | ...............................******.......................***.........****X | 62.800 | | 49.800 | .................................................**X | 49.800 | | 46.000 | .............................................*****X | 46.000 | | 22.300 | .....................*X | 22.300 | | 15.200 | ..............*****X | 15.200 | | 3.500 | ..*X | 3.500 | | 2.200 | .******............................**.................******....................*X | 65.700 | | 50.600 | ..................................................*****X | 50.600 | | 9.300 | ........**.....******...........................................................................***.......****X | 94.800 | | 22.700 | ......................**X | 22.700 | | 9.600 | ........*****X | 9.600 | | 2.200 | .********...........................**...*****X | 32.200 | | 24.800 | ........................*******X | 24.800 | | 5.700 | ....**X | 5.700 | | 0.000 | ********.******..................**......*****X | 25.100 | | 20.600 | ....................*X | 20.600 | | 13.200 | ............*****X | 13.200 | | 0.000 | **.******..........................***......****X | 32.900 | | 21.200 | ....................**X | 21.200 | | 15.000 | ..............********........******...................*X | 42.000 | | 38.500 | ......................................******.................***........****X | 63.200 | | 39.300 | .......................................**X | 39.300 | | 30.200 | .............................*****X | 30.200 | | 27.600 | ...........................**.******....**......******.......................*X | 61.000 | | 58.800 | ...........................................................*****X | 58.800 | | 16.000 | ...............*X | 16.000 | | 7.600 | ......******.......................***........****X | 38.200 | | 36.600 | ....................................***..******..*X | 40.400 | | 32.000 | ...............................*****X | 32.000 | | 23.000 | ......................**..*****X | 25.200 | | 13.600 | ............*X | 13.600 | | 11.000 | ..........******..........................................................***.............*****X | 80.400 | | 33.400 | .................................*X | 33.400 | | 24.600 | ........................******...........********..**..*****X | 40.000 | | 15.200 | ..............*X | 15.200 | | 7.600 | ......******........................***.........*****..............***..*****X | 55.200 | | 43.200 | ...........................................*X | 43.200 | | 37.800 | .....................................*****X | 37.800 | | 13.800 | .............*X | 13.800 | | 9.800 | ........******....................................................................**..................*****X | 93.400 | | 26.400 | .........................**X | 26.400 | | 13.400 | ............*****.....................................***..*****X | 51.200 | | 14.800 | ..............*X | 14.800 | | 5.800 | ....*****X | 5.800 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 12:48:25,164 | test | INFO | pool-9-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 11753.500 | ...................................................................................................X | 11753.500 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 12:48:26,453 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 12:48:37,854 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 12:48:52,016 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 12:48:52,016 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 12:49:02,815 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 12:49:02,815 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 12:49:13,698 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 12:49:13,698 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 12:49:32,062 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 46000 | 9.77 GiB | 207.66 MiB | 341.21 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 4.88 GiB | 461.25 MiB | 631.94 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 30021250 | 2.50 GiB | 1.60 GiB | 57.58 GiB | 24.1864779115 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 12:49:32,539 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 12:49:32,539 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 12:49:39,081 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 12:49:39,082 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 12:49:45,530 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 32% 2022-02-02 12:49:45,532 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 0% 2022-02-02 12:50:41,115 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 12:50:41,525 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 12:50:41,526 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 12:51:11,670 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 12:51:12,096 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 12:51:12,098 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 12:51:42,210 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 12:51:42,622 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 12:51:42,624 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 12:54:44,078 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'acf763387dcba7398eb9c70665ae5332', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a61ef58d810c3f9d7f011398fd04bc95', u'status': u'notRunning'} - rebalance failed 2022-02-02 12:54:44,104 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 12:54:44,104 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643835282833L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:54:42.833Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = a481e5ac820556eb63a23de955fb390b'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643835282832L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:54:42.832Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643835102828L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:51:42.828Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = a481e5ac820556eb63a23de955fb390b"} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643835102820L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T12:51:42.820Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643835092693L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:51:32.693Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643835076532L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:51:16.532Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643835074052L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:51:14.052Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.177', u'tstamp': 1643835063219L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:51:03.219Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643835058970L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:50:58.970Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 12:54:44,105 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643835011214L, u'shortText': u'message', u'serverTime': u'2022-02-02T12:50:11.214Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes 2022-02-02 12:54:44,107 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'acf763387dcba7398eb9c70665ae5332', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a61ef58d810c3f9d7f011398fd04bc95', u'status': u'notRunning'} - rebalance failed return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'acf763387dcba7398eb9c70665ae5332', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a61ef58d810c3f9d7f011398fd04bc95', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 4778.447s OK During the test, Remote Connections: 256, Disconnections: 256 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_6 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_7 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,data_load_stage=during,doc_size=512,randomize_value=True,skip_validations=False,GROUP=graceful_failover_and_delta_recovery_P0_set1' Test Input params: {'data_load_stage': 'during', 'doc_size': '512', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'step_count': '1', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_7', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_delta_recovery_P0_set1', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 7, 'randomize_value': 'True'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 12:54:49,926 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 12:54:50,319 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 12:55:53,970 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 63.3849999905 seconds to complete 2022-02-02 12:55:53,977 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 12:55:59,072 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x60ba55 type=client open_count=1 channel=[id: 0x74ac0626, 0.0.0.0/0.0.0.0:46054] timeout=300.0>) 2022-02-02 12:56:04,059 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x60ba56 type=client open_count=1 channel=[id: 0xec7bc002, 0.0.0.0/0.0.0.0:51804] timeout=300.0>) 2022-02-02 12:56:06,069 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x60ba57 type=client open_count=1 channel=[id: 0xc71245d7, 0.0.0.0/0.0.0.0:47740] timeout=300.0>) 2022-02-02 12:56:08,105 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x60ba58 type=client open_count=1 channel=[id: 0x8d8d87a1, 0.0.0.0/0.0.0.0:49381] timeout=300.0>) 2022-02-02 12:56:10,118 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x60ba59 type=client open_count=1 channel=[id: 0x924bba89, 0.0.0.0/0.0.0.0:53146] timeout=300.0>) 2022-02-02 12:56:12,128 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x60ba5a type=client open_count=1 channel=[id: 0x64c98488, 0.0.0.0/0.0.0.0:58290] timeout=300.0>) 2022-02-02 12:56:17,865 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x60ba5b type=client open_count=1 channel=[id: 0xc1ed18e2, 0.0.0.0/0.0.0.0:57637] timeout=300.0>) 2022-02-02 12:56:19,875 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x60ba5c type=client open_count=1 channel=[id: 0x5426bb3b, 0.0.0.0/0.0.0.0:33600] timeout=300.0>) 2022-02-02 12:56:21,887 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x60ba5d type=client open_count=1 channel=[id: 0x76944552, 0.0.0.0/0.0.0.0:60365] timeout=300.0>) 2022-02-02 12:56:23,898 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x60ba5e type=client open_count=1 channel=[id: 0x6d6c6eff, 0.0.0.0/0.0.0.0:38807] timeout=300.0>) 2022-02-02 12:56:29,260 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x60ba5f type=client open_count=1 channel=[id: 0x1e7f9105, 0.0.0.0/0.0.0.0:48181] timeout=300.0>) 2022-02-02 12:56:31,272 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x60ba60 type=client open_count=1 channel=[id: 0xabacb183, 0.0.0.0/0.0.0.0:46114] timeout=300.0>) 2022-02-02 12:56:33,282 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x60ba61 type=client open_count=1 channel=[id: 0xb27780c3, 0.0.0.0/0.0.0.0:47177] timeout=300.0>) 2022-02-02 12:56:35,293 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x60ba62 type=client open_count=1 channel=[id: 0xd5118311, 0.0.0.0/0.0.0.0:41570] timeout=300.0>) 2022-02-02 12:56:40,561 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x60ba63 type=client open_count=1 channel=[id: 0x28adc3bc, 0.0.0.0/0.0.0.0:60659] timeout=300.0>) 2022-02-02 12:56:42,572 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x60ba64 type=client open_count=1 channel=[id: 0x2d4d73a3, 0.0.0.0/0.0.0.0:56149] timeout=300.0>) 2022-02-02 12:56:44,582 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x60ba65 type=client open_count=1 channel=[id: 0xc89ab5d9, 0.0.0.0/0.0.0.0:47446] timeout=300.0>) 2022-02-02 12:56:46,592 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x60ba66 type=client open_count=1 channel=[id: 0xe9c95c0c, 0.0.0.0/0.0.0.0:53084] timeout=300.0>) 2022-02-02 12:56:52,150 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x60ba67 type=client open_count=1 channel=[id: 0x629ed7d2, 0.0.0.0/0.0.0.0:49201] timeout=300.0>) 2022-02-02 12:56:54,161 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x60ba68 type=client open_count=1 channel=[id: 0xcedfd6f4, 0.0.0.0/0.0.0.0:33354] timeout=300.0>) 2022-02-02 12:56:56,174 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x60ba69 type=client open_count=1 channel=[id: 0xa1ae5bbd, 0.0.0.0/0.0.0.0:56861] timeout=300.0>) 2022-02-02 12:56:58,187 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x60ba6a type=client open_count=1 channel=[id: 0x680aa788, 0.0.0.0/0.0.0.0:41016] timeout=300.0>) 2022-02-02 12:57:00,197 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x60ba6b type=client open_count=1 channel=[id: 0xfacf7f66, 0.0.0.0/0.0.0.0:52262] timeout=300.0>) 2022-02-02 12:57:05,716 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x60ba6c type=client open_count=1 channel=[id: 0x39f78d84, 0.0.0.0/0.0.0.0:56220] timeout=300.0>) 2022-02-02 12:57:07,726 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x60ba6d type=client open_count=1 channel=[id: 0x6169ed29, 0.0.0.0/0.0.0.0:33993] timeout=300.0>) 2022-02-02 12:57:09,739 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x60ba6e type=client open_count=1 channel=[id: 0x9169f4da, 0.0.0.0/0.0.0.0:41458] timeout=300.0>) 2022-02-02 12:57:11,750 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x60ba6f type=client open_count=1 channel=[id: 0x6707ac14, 0.0.0.0/0.0.0.0:59820] timeout=300.0>) 2022-02-02 12:57:17,180 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x60ba70 type=client open_count=1 channel=[id: 0xc7d1ab64, 0.0.0.0/0.0.0.0:58642] timeout=300.0>) 2022-02-02 12:57:19,190 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x60ba71 type=client open_count=1 channel=[id: 0x0f3fd193, 0.0.0.0/0.0.0.0:44748] timeout=300.0>) 2022-02-02 12:57:21,200 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x60ba72 type=client open_count=1 channel=[id: 0x51546993, 0.0.0.0/0.0.0.0:35185] timeout=300.0>) 2022-02-02 12:57:23,213 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 12:57:38,079 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 12:57:50,565 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 12:57:50,565 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 12:57:50,584 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 12:58:04,161 | test | INFO | pool-10-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.86413111473 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 12:58:04,180 | test | INFO | pool-10-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 12:58:19,217 | test | INFO | pool-10-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 12:58:19,230 | test | INFO | pool-10-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0690000057 sec 2022-02-02 12:58:19,354 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.551793328317 | 11.45 GiB | 10.65 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 1.06623181134 | 11.45 GiB | 10.65 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 0.965275166103 | 11.45 GiB | 10.66 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.237737737738 | 11.45 GiB | 10.72 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.25219133484 | 11.45 GiB | 10.55 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 12:58:19,355 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 12:58:19,355 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 12:58:19,545 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 12:58:19,997 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 12:58:20,328 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 89.53 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 12:58:20,329 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 12:58:29,523 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 12:58:29,532 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 13:26:10,181 | test | INFO | pool-10-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.................**...........******...............................................................**........*****X | 68168.500 | | 49456.200 | .......................................................................**X | 49456.200 | | 46391.500 | ..................................................................*****......***..*****X | 51665.600 | | 33686.100 | ................................................*X | 33686.100 | | 29919.200 | ..........................................******............**...******..........................**....*****X | 60942.800 | | 30018.400 | ..........................................*X | 30018.400 | | 22470.800 | ...............................******............**......******.............**.....*****............................***....****X | 68639.300 | | 60262.300 | ......................................................................................**X | 60262.300 | | 56354.800 | .................................................................................*****X | 56354.800 | | 28630.900 | ........................................*X | 28630.900 | | 23722.900 | .................................*****X | 23722.900 | | 20424.900 | ............................*X | 20424.900 | | 18844.700 | ..........................******..............**.......******........*X | 38761.500 | | 38433.400 | ......................................................****X | 38433.400 | | 29448.400 | .........................................**X | 29448.400 | | 27512.200 | .......................................****X | 27512.200 | | 24959.900 | ...................................**X | 24959.900 | | 23772.400 | .................................*****X | 23772.400 | | 22229.400 | ...............................**.******....*X | 25696.400 | | 25166.100 | ...................................*****X | 25166.100 | | 15309.300 | .....................*X | 15309.300 | | 13297.400 | ..................******............**....******...**X | 26200.900 | | 25401.800 | ....................................****X | 25401.800 | | 15739.900 | .....................**X | 15739.900 | | 14073.100 | ...................******.....*******X | 17789.700 | | 15285.100 | .....................*X | 15285.100 | | 15085.400 | ....................******...********.........**..******...*X | 26579.900 | | 26128.900 | .....................................*****X | 26128.900 | | 19070.900 | ..........................*X | 19070.900 | | 17961.600 | .........................********X | 18109.700 | | 18053.700 | .........................*****...***..*****.............********X | 30504.100 | | 24821.800 | ...................................*X | 24821.800 | | 23346.500 | .................................*****X | 23346.500 | | 15657.300 | .....................*X | 15657.300 | | 14808.700 | ....................******..*X | 15837.400 | | 15632.700 | .....................*****X | 15632.700 | | 12086.200 | ................**X | 12086.200 | | 11120.600 | ...............****X | 11120.600 | | 10195.500 | .............***.*****...........***.****X | 18671.700 | | 14156.600 | ...................**X | 14156.600 | | 13618.600 | ..................*****...........***..****X | 22536.200 | | 19366.100 | ...........................**X | 19366.100 | | 18419.800 | .........................*****..**X | 19515.200 | | 19339.000 | ...........................*****X | 19339.000 | | 18520.100 | .........................**.******....*******X | 21619.800 | | 15802.700 | ......................*X | 15802.700 | | 14984.300 | ....................******....**.******....**..*****X | 22072.100 | | 19983.000 | ............................*X | 19983.000 | | 19415.300 | ...........................******...*X | 21824.700 | | 21658.100 | ..............................*****X | 21658.100 | | 14657.700 | ....................**X | 14657.700 | | 13324.300 | ..................********.******...*X | 15893.600 | | 15279.000 | .....................*****X | 15279.000 | | 13636.300 | ..................**.******.........**..*****X | 21960.800 | | 19299.700 | ...........................**X | 19299.700 | | 18532.300 | .........................*****........***...****X | 25496.800 | | 25319.300 | ...................................**X | 25319.300 | | 24570.100 | ..................................*****X | 24570.100 | | 18884.000 | ..........................*X | 18884.000 | | 17889.100 | .........................******.********...**.******.....*******X | 25218.400 | | 15894.700 | ......................*X | 15894.700 | | 14188.400 | ...................******.***.*****....***.****X | 18897.800 | | 18369.000 | .........................**X | 18369.000 | | 18260.200 | .........................******......**...*****X | 24249.600 | | 21046.800 | .............................*X | 21046.800 | | 18526.100 | .........................*****X | 18526.100 | | 13336.200 | ..................*X | 13336.200 | | 12245.900 | ................*****X | 12245.900 | | 11769.800 | ................**..******..........**.*******X | 21006.800 | | 20975.300 | .............................*****X | 20975.300 | | 17163.200 | ........................*X | 17163.200 | | 16856.500 | .......................******.....**.*****X | 20693.400 | | 16145.100 | ......................**X | 16145.100 | | 15433.300 | .....................*****.........***.****X | 22539.400 | | 16368.200 | ......................**X | 16368.200 | | 14789.400 | ....................*****X | 14789.400 | | 13263.800 | ..................*X | 13263.800 | | 13137.900 | ..................******...............**....*****X | 26703.000 | | 23734.300 | .................................*X | 23734.300 | | 21824.900 | ..............................*****X | 21824.900 | | 15799.800 | ......................*X | 15799.800 | | 15477.100 | .....................******......**.*****X | 20359.500 | | 20038.600 | ............................*X | 20038.600 | | 19668.700 | ...........................*****X | 19668.700 | | 16249.600 | ......................*X | 16249.600 | | 15820.000 | ......................******.***.****X | 17537.700 | | 16724.500 | .......................**X | 16724.500 | | 15779.800 | .....................****X | 15779.800 | | 13633.200 | ..................**X | 13633.200 | | 13491.300 | ..................******..**.******........**..*****X | 22455.300 | | 19933.800 | ............................*X | 19933.800 | | 18328.200 | .........................*****X | 18328.200 | | 13429.000 | ..................*X | 13429.000 | | 13274.700 | ..................******..........**..*****X | 21479.100 | | 19970.000 | ............................**X | 19970.000 | | 19183.000 | ..........................****X | 19183.000 | | 17327.000 | ........................*********.*X | 18495.700 | | 18351.000 | .........................*****X | 18351.000 | | 14107.100 | ...................*X | 14107.100 | | 13007.800 | .................******......**..*****X | 18282.400 | | 15613.100 | .....................*X | 15613.100 | | 15217.000 | .....................******..**X | 17095.300 | | 16846.000 | .......................****X | 16846.000 | | 14029.400 | ...................**X | 14029.400 | | 13469.000 | ..................******...**..******...*X | 19202.100 | | 18146.100 | .........................******..**.*****X | 20584.600 | | 16019.100 | ......................*X | 16019.100 | | 14993.300 | ....................*****X | 14993.300 | | 14460.000 | ....................*X | 14460.000 | | 13124.200 | ..................********.*****X | 14214.700 | | 12876.800 | .................*X | 12876.800 | | 12354.200 | ................******......***..****X | 17333.000 | | 15385.000 | .....................**X | 15385.000 | | 14691.200 | ....................*****X | 14691.200 | | 13939.100 | ...................*******X | 13962.500 | | 10856.600 | ..............*X | 10856.600 | | 9944.100 | .............******.............**..********.*****X | 20600.200 | | 13959.200 | ...................*X | 13959.200 | | 12514.700 | .................*****X | 12514.700 | | 11863.400 | ................********.......***..*****X | 18055.700 | | 17315.700 | ........................*X | 17315.700 | | 16647.700 | .......................******.*******X | 17540.000 | | 15254.600 | .....................*X | 15254.600 | | 14830.200 | ....................*****X | 14830.200 | | 13239.200 | ..................**X | 13239.200 | | 12316.600 | ................*****....***..******...**.*****X | 18702.600 | | 10973.000 | ..............*X | 10973.000 | | 8428.800 | ...........******...**.******...........**...*****X | 20911.400 | | 18274.500 | .........................*X | 18274.500 | | 17545.700 | ........................******...*******X | 19534.700 | | 15580.400 | .....................**X | 15580.400 | | 14417.900 | ....................*****X | 14417.900 | | 12444.800 | .................*X | 12444.800 | | 12214.400 | ................******.....********.....**...*****X | 20593.800 | | 14263.700 | ...................*X | 14263.700 | | 11697.800 | ................*****X | 11697.800 | | 11463.300 | ...............********....***..*******X | 15614.100 | | 15478.900 | .....................*****...**X | 17193.800 | | 17144.800 | .......................******.**.******..*X | 19846.300 | | 19550.800 | ...........................*****X | 19550.800 | | 14911.400 | ....................*X | 14911.400 | | 14478.200 | ....................******........**...*****X | 22101.900 | | 17855.900 | .........................**X | 17855.900 | | 15685.500 | .....................****X | 15685.500 | | 13497.500 | ..................*********...**..******....*X | 19282.000 | | 18710.900 | ..........................*****X | 18710.900 | | 17515.100 | ........................********...*******X | 19867.400 | | 13769.700 | ...................*X | 13769.700 | | 12330.500 | ................******......***.*****...***..*****X | 19917.400 | | 18154.300 | .........................*X | 18154.300 | | 16840.900 | .......................********.*****X | 17361.600 | | 15680.900 | .....................*X | 15680.900 | | 15291.900 | .....................******.**X | 16351.500 | | 15895.200 | ......................****X | 15895.200 | | 13246.600 | ..................***.******..........**.*****X | 21531.500 | | 15865.600 | ......................*X | 15865.600 | | 14839.900 | ....................******.......**..*****X | 20712.100 | | 11474.200 | ...............*X | 11474.200 | | 8770.600 | ...........******......***...*******X | 15080.200 | | 13453.400 | ..................******.....**...*****X | 18913.800 | | 18026.500 | .........................*X | 18026.500 | | 17737.800 | ........................******.**X | 17903.900 | | 17358.600 | ........................****X | 17358.600 | | 16715.200 | .......................********X | 16882.100 | | 15668.700 | .....................*X | 15668.700 | | 15163.700 | .....................*****X | 15163.700 | | 14910.500 | ....................**.**X | 15422.500 | +-----------+----------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 13:26:10,825 | test | INFO | pool-10-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.......................**.....*****X | 150.000 | | 29.100 | ....**X | 29.100 | | 0.000 | **********************************************.....................***........*****...**X | 168.000 | | 148.600 | ...........................****X | 148.600 | | 19.400 | ..**X | 19.400 | | 0.000 | *****************************************************...............................***.......*****X | 200.000 | | 38.800 | ......*X | 38.800 | | 0.000 | *************************************************************************************************************************.............................................................................**.......................*****X | 519.400 | | 237.300 | ............................................**X | 237.300 | | 130.600 | ........................*****X | 130.600 | | 9.700 | *X | 9.700 | | 0.000 | ************************************************************************************************************************.......***..****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ******......................................**..........*****X | 250.000 | | 88.800 | ................*X | 88.800 | | 50.000 | ........*****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | ************************************************************************************************************************......................................**..........*****X | 250.000 | | 48.500 | ........*X | 48.500 | | 9.700 | ******..............................................................**............*****X | 390.300 | | 67.900 | ............*X | 67.900 | | 0.000 | **********************************************************************************************************************************...............**...............******......................................................*X | 440.300 | | 420.900 | ................................................................................*****X | 420.900 | | 179.100 | .................................*X | 179.100 | | 120.900 | ......................*****X | 120.900 | | 40.300 | ......**.....******.......**X | 103.000 | | 87.300 | ...............****X | 87.300 | | 6.700 | **X | 6.700 | | 0.000 | *********************************************************************************************************************************...............***.......******................................**.....*****X | 309.700 | | 261.300 | .................................................*X | 261.300 | | 232.200 | ...........................................******...................**..*****X | 338.700 | | 88.800 | ................*X | 88.800 | | 50.000 | ........*****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | **************************************************************************************************************************************************...............................**.......*****X | 200.000 | | 119.400 | .....................**X | 119.400 | | 100.000 | ..................****X | 100.000 | | 19.400 | ..**X | 19.400 | | 0.000 | *****************************************************************************************************************************************...***.******..*X | 34.400 | | 24.700 | ...******..........................................**...........*****X | 300.000 | | 179.100 | .................................**X | 179.100 | | 150.000 | ...........................****X | 150.000 | | 29.100 | ....**X | 29.100 | | 0.000 | ********************************************************************************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 13:26:11,996 | test | INFO | pool-10-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 322.400 | .....................................................................**.................*****X | 400.000 | | 77.600 | ...............**X | 77.600 | | 0.000 | ***********************************************..............................................................................**...................*****X | 450.000 | | 208.200 | ............................................*X | 208.200 | | 150.000 | ...............................*****X | 150.000 | | 29.100 | .....*X | 29.100 | | 0.000 | ******************************************************.............................................................***.................*****X | 359.700 | | 117.900 | ........................*X | 117.900 | | 40.300 | .......*****X | 40.300 | | 0.000 | ***********************************************************************************************************..........................**......******.....................................................**...............*****X | 459.700 | | 137.300 | ............................*X | 137.300 | | 40.300 | .......*****X | 40.300 | | 0.000 | ***************************************************************************************************************************........***..****X | 50.000 | | 9.700 | .**X | 9.700 | | 0.000 | ******......................................................................**.................*****X | 400.000 | | 117.900 | ........................*X | 117.900 | | 50.000 | .........*****X | 50.000 | | 9.700 | .**X | 9.700 | | 0.000 | ************************************************************************************************************************........**..********..******..................*******X | 140.300 | | 19.400 | ...*X | 19.400 | | 0.000 | ************************************************************************************************************************************........******......................................................................*******X | 361.200 | | 159.700 | .................................**X | 159.700 | | 150.000 | ...............................****X | 150.000 | | 29.100 | .....**X | 29.100 | | 0.000 | **************************************************************************************************************************************************........................**.......******...........***......*****X | 224.400 | | 217.200 | ..............................................*X | 217.200 | | 182.800 | ......................................*****X | 182.800 | | 61.900 | ............*X | 61.900 | | 50.000 | .........*****X | 50.000 | | 9.700 | .*X | 9.700 | | 0.000 | ***********************************************************************************************************************************************************...........................................**...........*****X | 250.000 | | 88.800 | ..................*X | 88.800 | | 69.400 | ..............******.................**X | 150.000 | | 130.600 | ...........................****X | 130.600 | | 9.700 | .*********........*X | 50.000 | | 40.300 | .......*****X | 40.300 | | 0.000 | *******************************************************************************************************************.................***......*****X | 109.700 | | 69.400 | ..............*X | 69.400 | | 59.700 | ...........******...........................***********....****X | 200.000 | | 38.800 | .......**X | 38.800 | | 0.000 | *********************************************************************************************X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 13:26:13,217 | test | INFO | pool-10-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 15422.500 | ...................................................................................................X | 15422.500 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 13:26:13,378 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 43.1317273193 | 11.45 GiB | 8.51 GiB | 52.50 MiB / 3.50 GiB | 6512348 / 13100231 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 35.6741573034 | 11.45 GiB | 8.79 GiB | 24.00 MiB / 3.50 GiB | 6510142 / 13024397 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 47.9964038017 | 11.45 GiB | 8.49 GiB | 28.14 MiB / 3.50 GiB | 6502222 / 13002307 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 45.648815654 | 11.45 GiB | 8.43 GiB | 0.0 Byte / 3.50 GiB | 6527036 / 13032044 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 28.7536675596 | 11.45 GiB | 8.71 GiB | 0.0 Byte / 3.50 GiB | 6451796 / 12853893 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 13:26:22,010 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 13:26:36,243 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 13:26:36,243 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 13:26:47,138 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 13:26:47,138 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 13:26:58,042 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 13:26:58,042 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 13:27:16,503 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+--------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+--------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 250.63 MiB | 359.76 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 530.34 MiB | 540.40 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.65 GiB | 52.95 GiB | 18.654133538 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+--------------+ 2022-02-02 13:27:16,503 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 13:27:20,816 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_recovery 2022-02-02 13:27:20,818 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_recovery 2022-02-02 13:27:20,818 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:701] failing over nodes [ip:172.23.105.36 port:8091 ssh_username:root] 2022-02-02 13:27:50,118 | test | INFO | pool-10-thread-3 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 29.1589999199 seconds to complete 2022-02-02 13:27:50,121 | test | INFO | pool-10-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 13:32:55,122 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 13:32:55,124 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 13:32:55,134 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 13:44:24,743 | test | INFO | pool-10-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........********...........................................********.................****************....********.........................*******X | 48553.000 | | 41190.400 | .................................................................................********.*******X | 41662.800 | | 38795.800 | .............................................................................*******X | 38795.800 | | 35937.600 | .......................................................................********.*******X | 36406.400 | | 23212.400 | .............................................********.............********.................********......********.....********.............*******X | 49669.700 | | 42082.500 | ...................................................................................********............*******X | 47944.300 | | 46245.800 | ............................................................................................*******X | 46245.800 | | 38100.700 | ...........................................................................*******X | 38100.700 | | 36619.900 | ........................................................................X | 36619.900 | | 35059.600 | .....................................................................X | 35059.600 | | 35059.600 | .....................................................................****X | 35059.600 | | 35059.600 | .....................................................................**.*******X | 35341.800 | | 27733.800 | ......................................................*******X | 27733.800 | | 25928.900 | ...................................................********.........................*******X | 38646.400 | | 32952.500 | .................................................................********.......*******X | 36305.400 | | 21354.000 | .........................................********................................*******X | 36964.800 | | 27704.900 | ......................................................*******X | 27704.900 | | 26186.600 | ...................................................********..........********..........********.********.....********..................*******X | 47685.900 | | 41435.400 | ..................................................................................*******X | 41435.400 | | 36147.700 | .......................................................................*******X | 36147.700 | | 32240.700 | ...............................................................*******X | 32240.700 | | 28503.500 | ........................................................********....********..*******X | 31697.900 | | 29305.600 | ..........................................................*******X | 29305.600 | | 28029.100 | .......................................................********..*******X | 29063.500 | | 28797.000 | ........................................................********.........*******X | 32816.000 | | 30016.100 | ...........................................................*******X | 30016.100 | | 28074.500 | .......................................................********..............*******X | 34840.900 | | 28386.500 | ........................................................****************.********....********..*******X | 31852.500 | | 29339.000 | ..........................................................********X | 29339.000 | | 24291.400 | ...............................................********........********...*******X | 29592.800 | | 29230.300 | .........................................................*******X | 29230.300 | | 24439.400 | ................................................********..........*******X | 29575.100 | | 24257.100 | ...............................................*********........*******X | 27816.500 | | 27075.600 | .....................................................*******X | 27075.600 | | 27026.500 | .....................................................********.....*******X | 29678.400 | | 21282.200 | .........................................X | 21282.200 | +-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 13:44:24,898 | test | INFO | pool-10-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*******X | 168.000 | | 8.000 | ...********.............................*******X | 56.000 | | 24.000 | .............********........................********.......................*******X | 104.000 | | 64.000 | .....................................*******X | 64.000 | | 40.000 | ......................********...................*******X | 72.000 | | 32.000 | ..................********.......................*******X | 72.000 | | 24.000 | .............*********.........********.............................*******X | 88.000 | | 76.000 | ............................................*******X | 76.000 | | 52.000 | .............................********.........*******X | 66.000 | | 48.000 | ...........................*********...............................*******X | 100.000 | | 8.000 | ...********.........................********.........********.......*******X | 76.000 | | 34.000 | ...................*******X | 34.000 | | 26.000 | ..............*********.......................*******X | 64.000 | | 58.000 | .................................*******X | 58.000 | | 56.000 | ................................********.......********..*******X | 72.000 | | 18.000 | .........********...............********..*******X | 46.000 | | 10.000 | ....********.............................................................*******X | 112.000 | | 33.000 | ..................********.........................*******X | 75.000 | | 24.000 | .............*******X | 24.000 | | 20.000 | ..........********..................................................********X | 104.000 | | 50.000 | ............................*******X | 50.000 | | 30.000 | ................********...........................*******X | 74.000 | | 62.000 | ...................................*******X | 62.000 | | 20.000 | ..........********.....********.................*******X | 56.000 | | 34.000 | ...................********......................*******X | 72.000 | | 52.000 | .............................*******X | 52.000 | | 38.000 | .....................*********........*******X | 52.000 | | 24.000 | .............********..********................********..........................*******X | 98.000 | | 24.000 | .............********.....*******X | 32.000 | | 8.000 | ...*********..................................*******X | 64.000 | | 8.000 | ...****************...................********.....*******X | 48.000 | | 40.000 | ......................********..........********X | 56.000 | | 18.000 | .........********....*******X | 24.000 | | 0.000 | X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 13:44:25,227 | test | INFO | pool-10-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............................................*******X | 64.000 | | 0.000 | ********...........********......*******X | 24.000 | | 8.000 | ....********..............................*******X | 48.000 | | 40.000 | ............................********...............................*******X | 82.000 | | 40.000 | ............................********X | 40.000 | | 16.000 | ..........********..............................*******X | 56.000 | | 48.000 | ..................................*******X | 48.000 | | 16.000 | ..........********..............................********.............................*******X | 96.000 | | 58.000 | .........................................*********...................................*******X | 106.000 | | 66.000 | ...............................................********...............................*******X | 108.000 | | 32.000 | ......................********......*******X | 40.000 | | 28.000 | ...................********.........*******X | 40.000 | | 16.000 | ..........*********..............................********.*******X | 58.000 | | 8.000 | ....********...............................*******X | 50.000 | | 48.000 | ..................................********.........*******X | 60.000 | | 26.000 | ..................********.............********...................................................********X | 114.000 | | 0.000 | ********...................................................*******X | 70.000 | | 16.000 | ..........********...............................********............*******X | 74.000 | | 18.000 | ............********........................................................*******X | 94.000 | | 82.000 | ...........................................................*******X | 82.000 | | 71.300 | ...................................................X | 71.300 | | 36.000 | .........................********...................................*******X | 84.000 | | 18.000 | ............********.......*******X | 28.000 | | 26.000 | ..................*******X | 26.000 | | 20.000 | .............********...........................*******X | 56.000 | | 28.000 | ...................*******X | 28.000 | | 26.000 | ..................********..................................................*******X | 94.000 | | 44.000 | ...............................*********.........********....................*******X | 84.000 | | 30.000 | .....................********..............................................................................*******X | 136.000 | | 20.000 | .............********..................................*******X | 66.000 | | 18.000 | ............********.............................................********X | 80.000 | | 42.000 | .............................********..............*******X | 60.000 | | 18.000 | ............********............................*******X | 56.000 | | 42.000 | .............................********..........................................................*******X | 120.000 | | 56.000 | ........................................********..............********X | 76.000 | | 0.000 | X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 13:44:26,480 | test | INFO | pool-10-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 21282.200 | ...................................................................................................X | 21282.200 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 13:44:26,621 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 13:44:26,642 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 13:44:41,301 | test | INFO | pool-10-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 0.363454066926 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 17.6136363636 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 17.3208407192 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 18.2002022245 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 18.3316557625 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 13:44:41,319 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 13:44:51,355 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 13:45:01,397 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 57.87 2022-02-02 13:45:11,440 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:45:21,476 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:45:31,513 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:45:41,549 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:45:51,584 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:46:01,617 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:46:11,653 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:46:21,690 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 13:46:31,726 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.09 2022-02-02 13:46:41,763 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 60.46 2022-02-02 13:46:51,801 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 63.6 2022-02-02 13:47:01,845 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: running, progress: 71.81 2022-02-02 13:47:16,892 | test | INFO | pool-10-thread-2 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 13:47:16,914 | test | INFO | pool-10-thread-2 | [task:check:468] Rebalance completed with progress: 100% in 155.612999916 sec 2022-02-02 13:47:16,914 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:701] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 13:47:38,221 | test | INFO | pool-10-thread-6 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 21.1739997864 seconds to complete 2022-02-02 13:47:38,227 | test | INFO | pool-10-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 13:52:43,230 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 13:52:43,230 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 13:52:43,237 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 14:07:07,039 | test | INFO | pool-10-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..............***.........******...........................**....******...............**........******.....*X | 36287.700 | | 36282.800 | .................................................................................******........**.....*****X | 42251.800 | | 37547.300 | ...................................................................................*X | 37547.300 | | 34575.300 | .............................................................................********.*****X | 35316.700 | | 33691.000 | ...........................................................................*X | 33691.000 | | 33444.600 | ..........................................................................******.....**..*****X | 36277.600 | | 34750.400 | .............................................................................**X | 34750.400 | | 33407.000 | ..........................................................................****X | 33407.000 | | 27011.000 | ............................................................********.....................***..**X | 37343.100 | | 37343.100 | ...................................................................................*X | 37343.100 | | 33358.700 | ..........................................................................********.....**X | 35536.900 | | 35211.600 | ..............................................................................****X | 35211.600 | | 31923.600 | .......................................................................**X | 31923.600 | | 31473.400 | ......................................................................****X | 31473.400 | | 30752.700 | ....................................................................**X | 30752.700 | | 29718.500 | ..................................................................****X | 29718.500 | | 26834.200 | ...........................................................**X | 26834.200 | | 26833.400 | ...........................................................*****.............***....****X | 34381.500 | | 30156.600 | ...................................................................**X | 30156.600 | | 27925.900 | ..............................................................****X | 27925.900 | | 27837.000 | ..............................................................***..*******X | 29140.500 | | 28463.800 | ...............................................................*****X | 28463.800 | | 26206.900 | ..........................................................*X | 26206.900 | | 25401.700 | ........................................................*****X | 25401.700 | | 25167.700 | .......................................................**...******............**...******.........***.....*****..........***..****X | 44179.900 | | 42758.100 | ...............................................................................................**X | 42758.100 | | 41307.400 | ............................................................................................*X | 41307.400 | | 41307.400 | ............................................................................................**X | 41307.400 | | 37185.300 | ...................................................................................**X | 37185.300 | | 35230.500 | ..............................................................................*****X | 35230.500 | | 24668.300 | ......................................................*X | 24668.300 | | 22850.000 | ..................................................******...............**...*****X | 30874.400 | | 28123.400 | ..............................................................**.******...**X | 29606.300 | | 28193.100 | ..............................................................****X | 28193.100 | | 28146.800 | ..............................................................***..*****....***..*****X | 31696.200 | | 25638.100 | .........................................................*X | 25638.100 | | 22959.700 | ..................................................******....**..*****X | 25268.100 | | 25236.500 | ........................................................**....******.................*X | 34617.500 | | 33355.700 | ..........................................................................*****X | 33355.700 | | 22445.800 | .................................................*X | 22445.800 | | 20846.400 | ..............................................******.********..*X | 22245.000 | | 21586.800 | ...............................................*****X | 21586.800 | | 20978.500 | ..............................................**..******........***...*****....***...*****......********X | 32492.000 | | 27949.800 | ..............................................................*X | 27949.800 | | 26727.800 | ...........................................................*****X | 26727.800 | | 22328.700 | .................................................*X | 22328.700 | | 20640.200 | .............................................******..**.*****X | 21766.700 | | 20475.400 | .............................................**X | 20475.400 | | 20206.500 | ............................................****X | 20206.500 | | 18238.500 | ........................................**X | 18238.500 | | 17276.700 | ......................................*****X | 17276.700 | | 16387.500 | ....................................********......**..*****X | 20149.500 | | 17916.800 | .......................................**X | 17916.800 | | 17158.300 | .....................................****X | 17158.300 | | 16894.600 | .....................................**X | 16894.600 | | 16685.400 | ....................................******...**.******...**.*****X | 19883.400 | | 18302.100 | ........................................********......*******X | 21189.500 | | 16479.700 | ....................................**X | 16479.700 | | 14636.200 | ................................******.....**.....******..*X | 19912.800 | | 19503.000 | ...........................................******..*X | 20663.200 | | 20405.000 | .............................................******....**...*****X | 23830.800 | | 19426.400 | ..........................................**X | 19426.400 | | 17788.400 | .......................................*****.....***.*****X | 20664.200 | | 20619.500 | .............................................**.******.*X | 21443.700 | | 21322.600 | ...............................................*****X | 21322.600 | | 20650.800 | .............................................*X | 20650.800 | | 20590.700 | .............................................*****X | 20590.700 | | 19815.400 | ...........................................*X | 19815.400 | | 19477.100 | ...........................................******..***.****X | 20822.000 | | 18426.900 | ........................................**X | 18426.900 | | 16611.800 | ....................................*****X | 16611.800 | | 15047.600 | .................................**..******........*******X | 19856.200 | | 18488.300 | ........................................*X | 18488.300 | | 17914.700 | .......................................*****X | 17914.700 | | 16249.600 | ...................................**.******........**..*****X | 20778.700 | | 17072.800 | .....................................**X | 17072.800 | | 15815.000 | ..................................******....**...******..*X | 19703.100 | | 18738.000 | .........................................*****X | 18738.000 | | 17875.600 | .......................................*******X | 18063.800 | | 15492.000 | ..................................**X | 15492.000 | | 14877.700 | ................................*****......***..*****X | 18356.500 | | 16503.200 | ....................................*X | 16503.200 | | 15408.600 | .................................*****X | 15408.600 | | 15326.000 | .................................**..******....*******X | 17906.400 | | 16479.400 | ....................................**X | 16479.400 | +-----------+-------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 14:07:07,555 | test | INFO | pool-10-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................**........*****X | 38.000 | | 11.500 | .......*X | 11.500 | | 0.000 | ******............................**...............*****X | 57.800 | | 31.300 | ......................*X | 31.300 | | 19.800 | ..............******................**.*******X | 43.300 | | 41.000 | ..............................*****X | 41.000 | | 35.700 | ..........................***....................*****.................................**X | 105.700 | | 71.200 | ....................................................*****X | 71.200 | | 34.100 | ........................**.............******........................**.....*****X | 88.900 | | 41.200 | ..............................*X | 41.200 | | 22.800 | ................*********.................*****........................................................**X | 120.000 | | 103.900 | .............................................................................****X | 103.900 | | 61.500 | .............................................***.....****X | 68.400 | | 27.400 | ...................**X | 27.400 | | 9.600 | ......****X | 9.600 | | 8.200 | .....***......*****........................***......X | 55.500 | | 55.500 | .........................................****.........**X | 67.500 | | 58.900 | ...........................................*****X | 58.900 | | 9.800 | ......*X | 9.800 | | 0.000 | ******..................................**................******..*X | 69.600 | | 68.400 | ..................................................*****X | 68.400 | | 26.000 | ..................***..*****.....................................................................**X | 119.800 | | 115.800 | ......................................................................................*****X | 115.800 | | 46.900 | ..................................*X | 46.900 | | 40.000 | .............................******.....**......******.................*X | 77.300 | | 76.800 | .........................................................*****X | 76.800 | | 59.600 | ............................................**.******............**.*****X | 78.200 | | 29.300 | .....................**X | 29.300 | | 6.700 | ....******.............**................******......*X | 53.500 | | 39.600 | .............................*****X | 39.600 | | 35.400 | .........................**...*****X | 38.300 | | 30.500 | ......................*X | 30.500 | | 25.900 | ..................******.................***...................****X | 72.800 | | 62.200 | ..............................................X | 62.200 | | 56.500 | .........................................*X | 56.500 | | 29.300 | .....................*****X | 29.300 | | 9.600 | ......********........**.............******....................................*X | 84.600 | | 75.300 | ........................................................*****X | 75.300 | | 37.000 | ...........................*X | 37.000 | | 36.400 | ..........................*****X | 36.400 | | 28.300 | ....................*X | 28.300 | | 24.800 | .................*****X | 24.800 | | 7.500 | ....**X | 7.500 | | 0.000 | *****.........................................***..................*****X | 78.400 | | 55.900 | .........................................*X | 55.900 | | 46.000 | .................................******..**.........******....................................*X | 107.200 | | 106.300 | ...............................................................................*****X | 106.300 | | 31.600 | ......................**X | 31.600 | | 11.900 | ........*****.......***.*****X | 22.500 | | 10.500 | ......*X | 10.500 | | 7.600 | ....******.**...******........**...*****X | 26.700 | | 25.200 | ..................**X | 25.200 | | 20.900 | ..............*****.......***.****X | 31.500 | | 23.900 | .................X | 23.900 | | 19.600 | .............*X | 19.600 | | 17.200 | ............******............................................................**...........................*****X | 132.000 | | 50.900 | .....................................*X | 50.900 | | 18.100 | ............*****X | 18.100 | | 14.400 | .........*X | 14.400 | | 6.700 | ....*****X | 6.700 | | 0.000 | ***....*****..................********X | 29.100 | | 18.700 | .............********...........**.......******.......*X | 52.600 | | 50.700 | .....................................*****X | 50.700 | | 35.800 | ..........................**X | 35.800 | | 32.400 | .......................****X | 32.400 | | 15.300 | ..........**X | 15.300 | | 7.600 | ....******..........**.......*****X | 29.200 | | 15.800 | ..........*X | 15.800 | | 7.600 | ....******......................**..........*****X | 49.600 | | 27.300 | ...................*X | 27.300 | | 14.800 | ..........********......******.............***..****X | 43.400 | | 41.900 | ..............................**X | 41.900 | | 37.600 | ...........................****X | 37.600 | | 36.200 | ..........................**X | 36.200 | | 33.200 | ........................*****X | 33.200 | | 12.300 | ........**......******..............................*X | 59.500 | | 52.300 | ......................................*****X | 52.300 | | 7.700 | ....*X | 7.700 | | 0.000 | ******...........***......*****....***..*****X | 30.400 | | 9.600 | ......*X | 9.600 | | 0.000 | ******...**..******........**....*****X | 22.800 | | 12.400 | ........*********............*X | 28.000 | | 25.600 | ..................******.................**...........*****X | 62.800 | | 19.700 | .............*X | 19.700 | | 7.200 | ....******............*********.............*******X | 40.400 | | 7.700 | ....*X | 7.700 | | 0.000 | X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 14:07:08,392 | test | INFO | pool-10-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***...............................***..............*****X | 53.200 | | 16.100 | ............*X | 16.100 | | 0.000 | ******......................**..........*****X | 38.000 | | 11.500 | ........*X | 11.500 | | 9.200 | ......******.............................................**........*****X | 71.400 | | 34.300 | ............................*X | 34.300 | | 20.500 | ................******......................**............................******....**X | 84.500 | | 47.700 | .......................................****X | 47.700 | | 37.100 | ..............................***..............*****..................***.......*****X | 83.600 | | 51.800 | ..........................................*X | 51.800 | | 38.000 | ...............................*****X | 38.000 | | 22.100 | .................**..................******...............................................................*X | 117.000 | | 94.000 | ..............................................................................*****X | 94.000 | | 41.000 | .................................***....****X | 45.600 | | 13.800 | ..........**X | 13.800 | | 2.300 | *****..............***........*****.............***.........*****...................**X | 75.700 | | 65.900 | ......................................................*****...............***...............*****X | 100.500 | | 51.400 | ..........................................*X | 51.400 | | 39.900 | ................................**X | 39.900 | | 39.900 | ................................***.............................**.........*****X | 84.600 | | 56.700 | ...............................................*X | 56.700 | | 48.700 | ........................................******....................................**X | 91.400 | | 80.400 | ...................................................................****X | 80.400 | | 20.400 | ................**X | 20.400 | | 15.200 | ...........******..............**............******.....*X | 51.600 | | 45.900 | .....................................*****X | 45.900 | | 43.400 | ...................................**.........******...........................*X | 84.900 | | 75.200 | ..............................................................*****X | 75.200 | | 12.800 | .........**X | 12.800 | | 0.000 | *****.................................***..............................*****.....................................**X | 117.800 | | 103.200 | ......................................................................................*****X | 103.200 | | 39.500 | ................................*X | 39.500 | | 23.900 | ...................*****X | 23.900 | | 22.400 | ..................**....******............................**...............*****X | 78.100 | | 51.700 | ..........................................**X | 51.700 | | 31.500 | .........................****X | 31.500 | | 26.200 | .....................**X | 26.200 | | 21.600 | .................******..**................******................................*X | 80.600 | | 79.300 | ..................................................................*****X | 79.300 | | 56.900 | ...............................................*X | 56.900 | | 43.500 | ...................................*****X | 43.500 | | 12.700 | .........*X | 12.700 | | 7.600 | .....******.........................**...........*****X | 49.600 | | 42.900 | ...................................**X | 42.900 | | 40.000 | ................................****X | 40.000 | | 24.200 | ...................**X | 24.200 | | 19.600 | ...............*****X | 19.600 | | 18.100 | ..............*X | 18.100 | | 12.800 | .........******.................**..............*****X | 48.700 | | 33.900 | ...........................*X | 33.900 | | 18.100 | ..............*****X | 18.100 | | 9.100 | ......***.**************.......................................**.................*****X | 76.400 | | 30.200 | ........................*X | 30.200 | | 9.600 | .......*************X | 9.600 | | 2.900 | .**X | 2.900 | | 0.000 | *****...........***.......******..................................**...........*****X | 74.700 | | 57.700 | ...............................................*X | 57.700 | | 54.800 | .............................................*****X | 54.800 | | 46.000 | ......................................*X | 46.000 | | 37.600 | ..............................*****X | 37.600 | | 20.300 | ................**X | 20.300 | | 17.600 | .............*****.............***.****X | 34.000 | | 27.100 | ......................X | 27.100 | | 15.400 | ............*X | 15.400 | | 9.600 | .......*****X | 9.600 | | 8.100 | .....*X | 8.100 | | 7.600 | .....******....................................**................*****X | 68.800 | | 43.500 | ...................................*X | 43.500 | | 32.400 | ..........................******.***...*****.**X | 38.800 | | 33.500 | ...........................*****X | 33.500 | | 8.200 | .....*X | 8.200 | | 0.000 | ******....................................................................**...............................*****X | 116.800 | | 35.700 | .............................*X | 35.700 | | 0.000 | ******....***..*****......***....*****..................***....****X | 45.200 | | 27.700 | ......................X | 27.700 | | 19.200 | ...............*X | 19.200 | | 12.000 | .........*****X | 12.000 | | 10.500 | .......**.......******.......................................*X | 64.300 | | 63.800 | .....................................................*****X | 63.800 | | 22.900 | ..................*X | 22.900 | | 12.800 | .........*****X | 12.800 | | 9.100 | ......***...******................**....*****X | 35.600 | | 14.800 | ...........*X | 14.800 | | 7.600 | .....*****************..*****....***..******.....********........................**.....*****X | 57.500 | | 24.800 | ....................**X | 24.800 | | 22.400 | ..................******....**....******..................*X | 53.200 | | 36.400 | .............................*****X | 36.400 | | 0.000 | ***X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 14:07:09,678 | test | INFO | pool-10-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 15479.600 | ...................................................................................................X | 15479.600 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 14:07:09,805 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 14:07:09,826 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 14:07:18,313 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 14:07:18,321 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 14:07:18,543 | test | INFO | pool-10-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 41.0682340915 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 21.5452818239 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 22.3910840932 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.600826135937 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 30.888719124 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 14:07:18,562 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 14:07:28,588 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 14:07:38,622 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 55.26 2022-02-02 14:07:48,664 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:07:58,706 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:08:08,749 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:08:18,789 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:08:28,831 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:08:38,875 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:08:48,918 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:08:58,954 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 14:09:09,007 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 60.4 2022-02-02 14:09:19,055 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 63.82 2022-02-02 14:09:29,091 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 66.36 2022-02-02 14:09:39,125 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 70.89 2022-02-02 14:09:49,164 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.65 2022-02-02 14:09:59,203 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.71 2022-02-02 14:10:09,240 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.71 2022-02-02 14:10:19,276 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.71 2022-02-02 14:10:29,313 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.71 2022-02-02 14:10:39,352 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.71 2022-02-02 14:10:49,398 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 72.71 2022-02-02 14:10:59,461 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 73.14 2022-02-02 14:11:09,500 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 77.18 2022-02-02 14:11:19,543 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 77.18 2022-02-02 14:11:29,582 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 77.18 2022-02-02 14:11:39,631 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 77.18 2022-02-02 14:11:49,681 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 77.37 2022-02-02 14:11:50,568 | test | ERROR | pool-10-thread-21 | [task:execute_tasks:3940] Failed to load 15 docs from 51077 to 75094 of thread_name LoadDocs_MutateDocsFromSpecTask_1643839643.51_default__default_mWuV6H-ep7ZbLRHPEXX2DThMUQV7tDKFfZMWm_-19-590000_ttl=0_delete__51077_75094_1643839643.93 2022-02-02 14:11:56,134 | test | ERROR | pool-10-thread-21 | [task:execute_tasks:3940] Failed to load 7 docs from 51077 to 75094 of thread_name LoadDocs_MutateDocsFromSpecTask_1643839643.51_default_gYtdRKkEHiGb0N0b-19-606000_Bo5bm5ASl25XFpwS9Txsxy9d6Ddup%HDiwqphhN83iv5NV9ZE4Fsc_9T8q6LE8RTUZ_i51IlHowX2qeLuclqvtYQILDXXTJHxOA3lnEMDh1ki5-HPH9354FyA36I6k58fO2rNlhQjbTThfg0nc3MvCW1oW4szHxQos4UYgfe38QhniI576wSsbPKnZgLNUy1ncBfE46ksFc_2PowPtgo3_PsgEI2ork-AW0-19-607000_ttl=0_delete__51077_75094_1643839643.93 2022-02-02 14:11:56,135 | test | ERROR | pool-10-thread-21 | [task:execute_tasks:3940] Failed to load 3 docs from 75094 to 94307 of thread_name LoadDocs_MutateDocsFromSpecTask_1643839643.51_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-19-609000_1jPqOK0lvRFVfgFiPd2H5at0pg5x6BCGpjb6rwpdoY%0oiXy9_bwxj2syelW-yUhElBvbOnAw3o_nr1g4sZu64Zk8qgpPrCP1OzaoLDGxeUteDw9-w6l5TOJWvYCjWXeYOSm7TgCkyHtYgbd5X%miLc8hfMucZ63MPU8xMH1DQYt3zNHValqNXmgPJ4Z4isskiut_%S9P8Dxv0u1tr%qB%%xeyXZl-bt_KP8x7K6__Ct-19-610000_ttl=0_read__75094_94307_1643839643.93 2022-02-02 14:11:56,137 | test | ERROR | pool-10-thread-21 | [task:execute_tasks:3940] Failed to load 12 docs from 75094 to 94307 of thread_name LoadDocs_MutateDocsFromSpecTask_1643839643.51_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-19-597000_Rxbn6qv%LVL_WPM%yqX5lWLX0UUJmjqO15IqEi7QaF1hWihtQwMMzkcy9cjRz18EIL6txwbos_sbv%2-19-599000_ttl=0_read__75094_94307_1643839643.94 2022-02-02 14:11:56,463 | test | ERROR | pool-10-thread-21 | [task:execute_tasks:3940] Failed to load 13 docs from 75094 to 94307 of thread_name LoadDocs_MutateDocsFromSpecTask_1643839643.51_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-19-603000_m_qyRM6E0FIBAx-T-b3b_P1OTtyYIu7qr1CXxgOAlGPMJm_Sz_-zUxFXM-8gYx8RZKk8t189dKY3IOhGpmeu%PnScEUK_1uiYxUv4djhRT-7S85DHPwPhRsEjuSpTM5w4qq%jjHW8vFkrmDaVJVMf%-Zh20c8usklVzSSMHNyKHDD0ex6EMnwMgcuMeHehCuy4sMRsrLsUtlEyK-19-603000_ttl=0_read__75094_94307_1643839643.94 2022-02-02 14:11:59,723 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 82.21 2022-02-02 14:12:07,505 | test | ERROR | pool-10-thread-21 | [task:execute_tasks:3940] Failed to load 11 docs from 51077 to 75094 of thread_name LoadDocs_MutateDocsFromSpecTask_1643839643.51_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-19-600000_SOG%U6-19-602000_ttl=0_delete__51077_75094_1643839643.95 2022-02-02 14:12:09,763 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: running, progress: 88.31 2022-02-02 14:12:24,818 | test | INFO | pool-10-thread-23 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 14:12:24,851 | test | INFO | pool-10-thread-23 | [task:check:468] Rebalance completed with progress: 100% in 306.308000088 sec 2022-02-02 14:22:23,157 | test | INFO | pool-10-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | 26.100 | ******..........................***.........*****......***.*****X | 16075.400 | | 12755.500 | ................................*X | 12755.500 | | 11662.200 | .............................******...**...******...................**...*****X | 22411.200 | | 20310.300 | ....................................................********...........*X | 24526.800 | | 24156.900 | ..............................................................*****X | 24156.900 | | 17328.100 | ............................................**X | 17328.100 | | 16679.400 | ..........................................*****...*X | 17885.000 | | 17877.500 | .............................................*****X | 17877.500 | | 17869.500 | .............................................**X | 17869.500 | | 17842.600 | .............................................*****..............***......****X | 25346.800 | | 21639.000 | .......................................................**X | 21639.000 | | 19992.800 | ...................................................*****X | 19992.800 | | 18996.000 | ................................................*X | 18996.000 | | 18852.500 | ................................................******..*X | 19824.600 | | 19617.800 | ..................................................******...**...******............**.*****X | 27003.500 | | 24856.000 | ................................................................*******X | 24981.500 | | 24767.500 | ...............................................................**X | 24767.500 | | 22960.800 | ...........................................................*****....***.....*****......**X | 28792.700 | | 28539.900 | .........................................................................****X | 28539.900 | | 25425.400 | .................................................................**X | 25425.400 | | 25012.900 | ................................................................****X | 25012.900 | | 24589.500 | ...............................................................**X | 24589.500 | | 24387.700 | ..............................................................****X | 24387.700 | | 23148.100 | ...........................................................**X | 23148.100 | | 23069.100 | ...........................................................*****...**X | 24339.800 | | 23682.600 | .............................................................*****X | 23682.600 | | 21141.500 | ......................................................********.....................**......*****X | 31639.600 | | 27812.700 | .......................................................................*X | 27812.700 | | 26705.700 | ....................................................................*****.....**X | 28255.200 | | 27901.300 | ........................................................................*****.X | 28378.100 | | 28378.100 | .........................................................................*X | 28378.100 | | 27652.600 | .......................................................................*****....................***........X | 38178.400 | | 38178.400 | ..................................................................................................*.X | 38178.400 | | 38178.400 | ..................................................................................................*.X | 38178.400 | | 33066.600 | .....................................................................................**X | 33066.600 | | 30123.600 | .............................................................................*****..***.....******........X | 35803.100 | | 35803.100 | ............................................................................................X | 35803.100 | | 34756.700 | ..........................................................................................*****X | 34756.700 | | 32971.100 | .....................................................................................X | 32971.100 | | 32971.100 | .....................................................................................X | 32971.100 | | 32290.300 | ...................................................................................*****X | 32290.300 | | 20813.200 | .....................................................*X | 20813.200 | | 19700.200 | ..................................................******...................**.*****X | 27299.300 | | 25578.900 | .................................................................*X | 25578.900 | | 25206.500 | .................................................................******.................**........******..*X | 35664.900 | | 34506.900 | .........................................................................................*****X | 34506.900 | | 31919.300 | ..................................................................................*X | 31919.300 | | 30945.400 | ................................................................................*****X | 30945.400 | | 30032.800 | .............................................................................**..******.....*X | 32523.300 | | 32208.200 | ...................................................................................*****X | 32208.200 | | 28071.800 | ........................................................................*X | 28071.800 | | 27106.700 | ......................................................................*****X | 27106.700 | | 24841.400 | ................................................................*X | 24841.400 | | 24611.600 | ...............................................................X | 24611.600 | | 24611.600 | ...............................................................**X | 24611.600 | | 24611.600 | ...............................................................**.......**..*****X | 27922.400 | | 27396.600 | ......................................................................**X | 27396.600 | | 26805.500 | .....................................................................****X | 26805.500 | | 21609.900 | .......................................................**X | 21609.900 | | 21259.100 | ......................................................*****....********.**X | 23037.300 | | 22460.100 | .........................................................*****X | 22460.100 | | 21027.500 | ......................................................**.*****X | 21445.700 | | 21441.600 | .......................................................*X | 21441.600 | | 21397.300 | .......................................................******...*X | 22662.500 | | 22426.600 | .........................................................*****X | 22426.600 | | 21740.600 | .......................................................**..******...*X | 23420.600 | | 22880.400 | ..........................................................*****X | 22880.400 | | 21261.900 | ......................................................*X | 21261.900 | | 20966.700 | .....................................................*****X | 20966.700 | | 20579.800 | ....................................................**.******.......**..*****X | 24219.300 | | 23268.900 | ...........................................................*X | 23268.900 | | 22412.100 | .........................................................*****X | 22412.100 | | 16761.700 | ..........................................**X | 16761.700 | | 16509.200 | ..........................................*****..................***...****X | 24725.200 | | 21596.400 | .......................................................**X | 21596.400 | | 20480.100 | ....................................................*****X | 20480.100 | | 16867.000 | ...........................................*X | 16867.000 | | 16525.300 | ..........................................******.............**...*****X | 22726.400 | | 21282.800 | ......................................................*X | 21282.800 | | 19692.700 | ..................................................*****X | 19692.700 | | 15250.200 | ......................................**..******.............**.******...*X | 22484.200 | | 22464.300 | .........................................................*****X | 22464.300 | | 20751.900 | .....................................................********.*X | 21309.600 | | 20878.500 | .....................................................*****X | 20878.500 | | 19765.600 | ..................................................*X | 19765.600 | | 19505.600 | ..................................................*****X | 19505.600 | | 18869.900 | ................................................********..***..*****......********X | 22611.800 | | 19188.400 | .................................................*X | 19188.400 | | 18640.400 | ...............................................******..**.******.......**.*****X | 22880.400 | | 22480.500 | .........................................................*X | 22480.500 | | 21903.700 | ........................................................*****X | 21903.700 | | 20134.600 | ...................................................*X | 20134.600 | | 20063.000 | ...................................................*****X | 20063.000 | | 17448.000 | ............................................*X | 17448.000 | | 17161.400 | ...........................................******...********..............***..*****X | 24292.100 | | 18077.900 | ..............................................*X | 18077.900 | | 16967.900 | ...........................................******...**.******..*X | 19360.600 | | 19272.300 | .................................................*****X | 19272.300 | | 18506.700 | ...............................................*X | 18506.700 | | 18478.900 | ...............................................*****X | 18478.900 | | 17238.400 | ............................................********...*X | 18530.600 | | 18426.800 | ...............................................******..**X | 19395.000 | | 19353.200 | .................................................****X | 19353.200 | | 18088.800 | ..............................................*******X | 18229.000 | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 14:22:23,855 | test | INFO | pool-10-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.........*****.........................................**X | 65.700 | | 53.100 | .......................................****X | 53.100 | | 0.000 | ********************.....*****.............................................................***...*****X | 91.300 | | 30.300 | ......................*X | 30.300 | | 18.300 | .............******........................**........*****X | 60.300 | | 16.300 | ...........*X | 16.300 | | 7.300 | ....*****X | 7.300 | | 1.200 | ********..................................................................***...........****X | 101.000 | | 21.700 | ...............**X | 21.700 | | 7.300 | ....******.................................**......*****X | 58.400 | | 15.700 | ...........*X | 15.700 | | 8.500 | .....******.....................................**..........*****X | 69.900 | | 46.800 | ..................................*X | 46.800 | | 33.000 | ........................*****X | 33.000 | | 1.200 | **X | 1.200 | | 0.000 | *****..............***....*****...............................***.......****X | 74.200 | | 25.400 | ..................**X | 25.400 | | 13.400 | .........******....**...******......................**......*****X | 59.800 | | 59.500 | ............................................*X | 59.500 | | 54.700 | ........................................*****X | 54.700 | | 44.400 | ................................*X | 44.400 | | 41.400 | ..............................*****X | 41.400 | | 4.800 | ..*X | 4.800 | | 1.200 | ******.................................**.....*****X | 49.900 | | 46.400 | ..................................**.*****X | 47.600 | | 28.000 | ....................*X | 28.000 | | 23.800 | .................******.............**....******...............*X | 65.800 | | 65.200 | ................................................*****X | 65.200 | | 28.600 | ....................*X | 28.600 | | 25.000 | ..................******..................**...........******....................................................*X | 130.700 | | 126.500 | ...............................................................................................*****X | 126.500 | | 33.300 | ........................**X | 33.300 | | 19.500 | .............*****...............***.....*****X | 45.700 | | 38.300 | ............................*X | 38.300 | | 36.500 | ..........................*****X | 36.500 | | 19.500 | .............*X | 19.500 | | 17.700 | ............******....*X | 22.500 | | 20.700 | ..............*****X | 20.700 | | 9.800 | ......**...******........................*X | 45.200 | | 43.400 | ................................*****X | 43.400 | | 16.400 | ...........*X | 16.400 | | 14.600 | ..........******.....**..........******.........................................**X | 88.000 | | 81.400 | .............................................................****X | 81.400 | | 21.300 | ...............**X | 21.300 | | 14.700 | ..........******...................**...........*****X | 53.600 | | 52.300 | .......................................*X | 52.300 | | 39.700 | .............................*****X | 39.700 | | 28.800 | .....................**...*****X | 33.000 | | 13.400 | .........*X | 13.400 | | 10.400 | ......******.....***.*****........................................***.......*****X | 79.400 | | 32.800 | ........................*X | 32.800 | | 22.600 | ................*****X | 22.600 | | 13.000 | ........**......******...................................*X | 65.700 | | 57.900 | ...........................................*****X | 57.900 | | 44.800 | .................................**......*****X | 53.200 | | 32.300 | .......................*X | 32.300 | | 29.300 | .....................******.*X | 30.600 | | 22.200 | ...............******..............***.......****X | 49.500 | | 38.600 | ............................***..******...*X | 44.500 | | 37.300 | ...........................*****X | 37.300 | | 17.700 | ............*X | 17.700 | | 17.100 | ............******......................**.....******..............***...****X | 75.200 | | 21.200 | ...............**X | 21.200 | | 7.400 | ....*****X | 7.400 | | 0.000 | ********......................................................**...........*****X | 86.000 | | 35.900 | ..........................*X | 35.900 | | 25.700 | ..................******...............**....******.*X | 52.000 | | 50.200 | .....................................*****X | 50.200 | | 8.400 | .....*X | 8.400 | | 3.000 | .******...............**X | 22.600 | | 20.800 | ..............****X | 20.800 | | 1.200 | ***.******......*X | 9.200 | | 7.400 | ....*****X | 7.400 | | 0.000 | ********...............................................**............*****X | 77.500 | | 22.500 | ................*X | 22.500 | | 6.900 | ....******.....**..*****X | 16.500 | | 15.200 | ..........**.******........................***...****X | 51.800 | | 39.600 | .............................**X | 39.600 | | 38.400 | ............................*****X | 38.400 | | 6.600 | ....*X | 6.600 | | 0.000 | ******......................**..........******....**X | 47.100 | | 33.900 | ........................****X | 33.900 | | 25.700 | ..................***.....******..................**..*****X | 58.400 | | 21.800 | ...............*X | 21.800 | | 15.800 | ...........******.........**...*******X | 31.600 | | 24.400 | .................*****X | 24.400 | | 0.000 | ***..*****....................................................***.....*****X | 77.900 | | 16.900 | ...........*X | 16.900 | | 6.100 | ...*****X | 6.100 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 14:22:24,572 | test | INFO | pool-10-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..................*****...........................................................*.......................*X | 100.800 | | 84.000 | ..................................................................................****X | 84.000 | | 21.000 | ...................***...*****X | 23.800 | | 4.200 | ...*X | 4.200 | | 0.000 | ******......**....******...........................................**....*****X | 58.400 | | 34.000 | ................................*X | 34.000 | | 25.600 | ........................******..............................***..........****X | 65.700 | | 23.000 | .....................**X | 23.000 | | 14.600 | .............****X | 14.600 | | 2.400 | .**X | 2.400 | | 1.200 | *****......................................................***........*****X | 64.500 | | 35.300 | ..................................*X | 35.300 | | 31.100 | .............................*****X | 31.100 | | 17.600 | ................*X | 17.600 | | 14.600 | .............*****X | 14.600 | | 8.500 | .......**.....******..........................................**.******........................***..****X | 82.800 | | 9.600 | ........**X | 9.600 | | 0.000 | *****.........................***.........*****...........**X | 45.600 | | 41.400 | ........................................*****.......*...................**........**X | 75.600 | | 75.600 | ..........................................................................*X | 75.600 | | 75.600 | ..........................................................................X | 75.600 | | 30.300 | .............................*X | 30.300 | | 23.100 | .....................******..........................**.......******...*X | 58.900 | | 56.500 | .......................................................*****X | 56.500 | | 46.200 | ............................................*X | 46.200 | | 40.200 | ......................................*****X | 40.200 | | 3.600 | ..*X | 3.600 | | 0.000 | ******...........................................**..........*****X | 54.200 | | 43.300 | .........................................*X | 43.300 | | 39.100 | .....................................*****X | 39.100 | | 18.200 | .................*X | 18.200 | | 15.800 | ..............*****............***...*****............................................***....****X | 79.600 | | 51.700 | ..................................................**X | 51.700 | | 50.700 | .................................................*****..*********...............*X | 67.700 | | 66.500 | ................................................................*****X | 66.500 | | 21.200 | ....................*X | 21.200 | | 14.600 | .............******.......**..******...........**..********.*****X | 38.300 | | 33.500 | ................................**X | 33.500 | | 29.300 | ............................*****......................................***...............*****.............**X | 96.400 | | 92.200 | ..........................................................................................*****X | 92.200 | | 27.300 | ..........................*X | 27.300 | | 15.300 | ..............******...................**.......******....*.............X | 59.100 | | 58.500 | .........................................................*****X | 58.500 | | 20.600 | ...................*X | 20.600 | | 16.400 | ...............******.................................................**................*****X | 81.800 | | 80.500 | ..............................................................................**X | 80.500 | | 67.900 | ..................................................................****X | 67.900 | | 47.000 | .............................................**X | 47.000 | | 44.600 | ...........................................****X | 44.600 | | 12.800 | ...........**X | 12.800 | | 9.200 | ........******.......**..******......**...*****X | 27.500 | | 20.100 | ..................*X | 20.100 | | 14.700 | .............******............**.....******............***..****X | 45.800 | | 42.800 | .........................................***.******....*******X | 48.200 | | 16.400 | ...............*X | 16.400 | | 7.400 | ......******................**........*****X | 31.300 | | 27.500 | ..........................*X | 27.500 | | 23.900 | ......................******.........***.....****X | 38.000 | | 14.500 | .............**X | 14.500 | | 6.100 | .....******........**.........******...................................**...*****X | 61.900 | | 31.000 | .............................*X | 31.000 | | 19.600 | ..................*********....*****.............................................***...........*****X | 79.900 | | 35.900 | ..................................*X | 35.900 | | 24.500 | .......................*****X | 24.500 | | 17.100 | ...............**.....******...................................................*******X | 73.300 | | 21.900 | ....................*X | 21.900 | | 17.700 | ................******.............***.****X | 31.800 | | 4.800 | ...**X | 4.800 | | 1.800 | ******....................*******X | 22.000 | | 14.600 | .............**...******.................................**...*****X | 54.300 | | 21.200 | ....................*X | 21.200 | | 14.000 | ............******.........**.*****X | 23.900 | | 9.100 | ........********..................*X | 28.000 | | 26.800 | .........................*****X | 26.800 | | 14.600 | .............**.******........********.....********X | 29.200 | | 17.000 | ...............*X | 17.000 | | 15.800 | ..............******....................................**.....*****X | 57.200 | | 38.900 | .....................................*X | 38.900 | | 37.700 | ....................................*****X | 37.700 | | 25.500 | ........................*X | 25.500 | | 20.700 | ...................*****X | 20.700 | | 14.600 | .............*******X | 14.600 | | 8.500 | .......**X | 8.500 | | 6.100 | .....******......**...******..............**....*****X | 34.100 | | 28.000 | ..........................*X | 28.000 | | 19.600 | ..................*****X | 19.600 | | 0.000 | X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 14:22:25,861 | test | INFO | pool-10-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 14301.400 | ...................................................................................................X | 14301.400 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 14:22:27,210 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 14:22:35,309 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 14:22:49,569 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 14:22:49,569 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 14:23:00,530 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 14:23:00,530 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 14:23:11,584 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 14:23:11,584 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 14:23:29,920 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 44100 | 9.77 GiB | 196.80 MiB | 342.24 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 44100 | 4.88 GiB | 433.14 MiB | 615.73 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 28820250 | 2.50 GiB | 1.63 GiB | 56.95 GiB | 32.1500264571 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 14:23:30,405 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 14:23:30,405 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 14:23:36,959 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 14:23:36,960 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 14:23:43,519 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 41% 2022-02-02 14:23:43,520 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 1% 2022-02-02 14:24:38,930 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 14:24:39,362 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 14:24:39,362 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 14:25:09,546 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 14:25:09,957 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 14:25:09,957 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 14:25:40,095 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 14:25:40,517 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 14:25:40,519 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 14:28:41,973 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7500cfa13581495ba3884228303eb5ce', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=febbee0806af70bfc024d0d3ea16c4ca', u'status': u'notRunning'} - rebalance failed 2022-02-02 14:28:42,000 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 14:28:42,000 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643840920732L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:28:40.732Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 807c76b412c4570220c4a27030e8d749'} 2022-02-02 14:28:42,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643840920731L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:28:40.731Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 14:28:42,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643840740727L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:25:40.727Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 807c76b412c4570220c4a27030e8d749"} 2022-02-02 14:28:42,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643840740719L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T14:25:40.719Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 14:28:42,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.177', u'tstamp': 1643840715709L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:25:15.709Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 14:28:42,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643840679852L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:24:39.852Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 14:28:42,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643840677978L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:24:37.978Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 14:28:42,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643840673263L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:24:33.263Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 14:28:42,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643840670576L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:24:30.576Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 14:28:42,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643840649023L, u'shortText': u'message', u'serverTime': u'2022-02-02T14:24:09.023Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance 2022-02-02 14:28:42,005 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7500cfa13581495ba3884228303eb5ce', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=febbee0806af70bfc024d0d3ea16c4ca', u'status': u'notRunning'} - rebalance failed progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7500cfa13581495ba3884228303eb5ce', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=febbee0806af70bfc024d0d3ea16c4ca', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 5637.899s OK During the test, Remote Connections: 271, Disconnections: 271 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 7 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_7 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_8 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,data_load_stage=during,doc_size=512,randomize_value=True,skip_validations=False,GROUP=hard_failover_and_delta_recovery_P0_set0' Test Input params: {'data_load_stage': 'during', 'doc_size': '512', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'step_count': '1', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_8', 'nodes_init': '5', 'GROUP': 'hard_failover_and_delta_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 8, 'randomize_value': 'True'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 14:28:47,811 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:28:48,194 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:29:51,848 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 63.3880000114 seconds to complete 2022-02-02 14:29:51,855 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 14:29:56,947 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x77d210 type=client open_count=1 channel=[id: 0xb247798a, 0.0.0.0/0.0.0.0:49977] timeout=300.0>) 2022-02-02 14:30:00,038 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x77d211 type=client open_count=1 channel=[id: 0x4a88cc77, 0.0.0.0/0.0.0.0:57015] timeout=300.0>) 2022-02-02 14:30:02,051 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x77d212 type=client open_count=1 channel=[id: 0xa5a221fb, 0.0.0.0/0.0.0.0:59064] timeout=300.0>) 2022-02-02 14:30:04,065 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x77d213 type=client open_count=1 channel=[id: 0x6cd0a80b, 0.0.0.0/0.0.0.0:47746] timeout=300.0>) 2022-02-02 14:30:06,075 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x77d214 type=client open_count=1 channel=[id: 0x92ad4fbf, 0.0.0.0/0.0.0.0:53069] timeout=300.0>) 2022-02-02 14:30:11,480 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x77d215 type=client open_count=1 channel=[id: 0xbd08b890, 0.0.0.0/0.0.0.0:43477] timeout=300.0>) 2022-02-02 14:30:13,490 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x77d216 type=client open_count=1 channel=[id: 0x298a27ff, 0.0.0.0/0.0.0.0:59343] timeout=300.0>) 2022-02-02 14:30:15,502 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x77d217 type=client open_count=1 channel=[id: 0xfc0aabc9, 0.0.0.0/0.0.0.0:44746] timeout=300.0>) 2022-02-02 14:30:17,513 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x77d218 type=client open_count=1 channel=[id: 0xcdc212b1, 0.0.0.0/0.0.0.0:44577] timeout=300.0>) 2022-02-02 14:30:23,178 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x77d219 type=client open_count=1 channel=[id: 0x959ed33c, 0.0.0.0/0.0.0.0:45157] timeout=300.0>) 2022-02-02 14:30:25,190 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x77d21a type=client open_count=1 channel=[id: 0x978c1f8b, 0.0.0.0/0.0.0.0:59904] timeout=300.0>) 2022-02-02 14:30:27,200 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x77d21b type=client open_count=1 channel=[id: 0x32840e96, 0.0.0.0/0.0.0.0:46078] timeout=300.0>) 2022-02-02 14:30:29,211 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x77d21c type=client open_count=1 channel=[id: 0x148bdceb, 0.0.0.0/0.0.0.0:34692] timeout=300.0>) 2022-02-02 14:30:34,392 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x77d21d type=client open_count=1 channel=[id: 0x1cde3f04, 0.0.0.0/0.0.0.0:51072] timeout=300.0>) 2022-02-02 14:30:36,403 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x77d21e type=client open_count=1 channel=[id: 0xc064f885, 0.0.0.0/0.0.0.0:35314] timeout=300.0>) 2022-02-02 14:30:38,414 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x77d21f type=client open_count=1 channel=[id: 0x10c2295d, 0.0.0.0/0.0.0.0:37236] timeout=300.0>) 2022-02-02 14:30:40,424 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x77d220 type=client open_count=1 channel=[id: 0x65ffb646, 0.0.0.0/0.0.0.0:33017] timeout=300.0>) 2022-02-02 14:30:45,871 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x77d221 type=client open_count=1 channel=[id: 0xdc1c34f3, 0.0.0.0/0.0.0.0:39452] timeout=300.0>) 2022-02-02 14:30:47,882 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x77d222 type=client open_count=1 channel=[id: 0xc7c76fb5, 0.0.0.0/0.0.0.0:46374] timeout=300.0>) 2022-02-02 14:30:49,892 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x77d223 type=client open_count=1 channel=[id: 0x06f1ab65, 0.0.0.0/0.0.0.0:58886] timeout=300.0>) 2022-02-02 14:30:51,904 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x77d224 type=client open_count=1 channel=[id: 0x1ad43118, 0.0.0.0/0.0.0.0:44062] timeout=300.0>) 2022-02-02 14:30:58,759 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x77d225 type=client open_count=1 channel=[id: 0x7c520614, 0.0.0.0/0.0.0.0:32861] timeout=300.0>) 2022-02-02 14:31:00,770 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x77d226 type=client open_count=1 channel=[id: 0x63d9b2c6, 0.0.0.0/0.0.0.0:39043] timeout=300.0>) 2022-02-02 14:31:02,780 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x77d227 type=client open_count=1 channel=[id: 0x8cf2ef35, 0.0.0.0/0.0.0.0:40663] timeout=300.0>) 2022-02-02 14:31:04,790 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x77d228 type=client open_count=1 channel=[id: 0x6b2f4d1b, 0.0.0.0/0.0.0.0:53278] timeout=300.0>) 2022-02-02 14:31:10,002 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x77d229 type=client open_count=1 channel=[id: 0x0d084321, 0.0.0.0/0.0.0.0:35545] timeout=300.0>) 2022-02-02 14:31:12,012 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x77d22a type=client open_count=1 channel=[id: 0x47514519, 0.0.0.0/0.0.0.0:53992] timeout=300.0>) 2022-02-02 14:31:14,022 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x77d22b type=client open_count=1 channel=[id: 0xa6370f43, 0.0.0.0/0.0.0.0:43725] timeout=300.0>) 2022-02-02 14:31:16,032 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 14:31:30,895 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 14:31:43,348 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 14:31:43,348 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:31:43,368 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:31:56,552 | test | INFO | pool-11-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.50489089541 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 14:31:56,571 | test | INFO | pool-11-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 14:32:11,608 | test | INFO | pool-11-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 14:32:11,621 | test | INFO | pool-11-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0699999332 sec 2022-02-02 14:32:11,740 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.401304238776 | 11.45 GiB | 10.65 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.575143785946 | 11.45 GiB | 10.67 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.49141496428 | 11.45 GiB | 10.60 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.387936428482 | 11.45 GiB | 10.71 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.09200451864 | 11.45 GiB | 10.55 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 14:32:11,740 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:32:11,740 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:32:11,907 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 14:32:12,388 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 14:32:12,709 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 85.43 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 14:32:12,709 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 14:32:23,082 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 14:32:23,091 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 14:58:42,476 | test | INFO | pool-11-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 194.000 | ..................******.................................................................................**X | 1000.000 | | 806.000 | ...............................................................................*****X | 806.000 | | 0.000 | *********************************************************............................**.......*****X | 350.000 | | 67.900 | .....*X | 67.900 | | 0.000 | ***********************************************************************........***....*****....**X | 169.400 | | 120.900 | ...........*****X | 120.900 | | 40.300 | ...**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *****************************************************************************************************************............**.....********X | 179.100 | | 120.900 | ...........*****X | 120.900 | | 0.000 | *********************************************************************************************************************************........***..*****.....*********.........*X | 240.300 | | 230.600 | ......................*****X | 230.600 | | 29.100 | .*X | 29.100 | | 0.000 | ***********************************************************************************************....***.*****................***....*****X | 250.000 | | 48.500 | ...*X | 48.500 | | 0.000 | ****************************************************************************************************************************************************.******.........********....................**........*****X | 388.800 | | 348.500 | .................................**X | 348.500 | | 241.800 | .......................*****X | 241.800 | | 0.000 | ***********************************************************************************************************....**.*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ***********************............**...*****X | 150.000 | | 109.700 | .........*X | 109.700 | | 100.000 | .........*****X | 100.000 | | 19.400 | *X | 19.400 | | 0.000 | *************************************************************************************************....**.******............**...*****X | 200.000 | | 79.100 | ......**X | 79.100 | | 69.400 | .....*****.........**X | 150.000 | | 140.300 | .............******....**..*****X | 200.000 | | 119.400 | ..........*X | 119.400 | | 90.300 | ........*****X | 90.300 | | 9.700 | **X | 9.700 | | 0.000 | ****************************************************************************************************************....***.******....**.******........**..*****X | 209.700 | | 129.100 | ...........*X | 129.100 | | 90.300 | ........*****X | 90.300 | | 9.700 | **X | 9.700 | | 0.000 | *****************************************************************************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 14:58:42,520 | test | INFO | pool-11-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...................******.................................................................................**X | 600.000 | | 483.600 | ...............................................................................****X | 483.600 | | 0.000 | *********************************************************...............................................**...........*****X | 350.000 | | 67.900 | ..........*X | 67.900 | | 0.000 | ************************************************************************.............**...........******..................................*X | 350.000 | | 301.500 | .................................................*****X | 301.500 | | 19.400 | ..*X | 19.400 | | 0.000 | *****************************************************************************************************************......***..*****.............................***.............*****X | 304.200 | | 233.100 | .....................................*X | 233.100 | | 145.800 | .......................*****X | 145.800 | | 0.000 | *************************************************************************************************************************************.*****........................................***.................******........................................*X | 590.300 | | 541.800 | .........................................................................................*****X | 541.800 | | 58.200 | ........*X | 58.200 | | 0.000 | ***********************************************************************************************.............***.....****X | 109.700 | | 69.400 | ..........**X | 69.400 | | 40.300 | .....********..*************X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | *********************************************************************************************************************************......***...*******X | 59.100 | | 40.300 | .....******......................................................**..................*****X | 469.400 | | 227.600 | ....................................**X | 227.600 | | 130.600 | ....................****X | 130.600 | | 9.700 | **X | 9.700 | | 0.000 | ********************************************************************************************************......***..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | **************.............***........*****....................**X | 250.000 | | 220.900 | ...................................****X | 220.900 | | 100.000 | ...............********X | 100.000 | | 19.400 | ..*X | 19.400 | | 0.000 | *************************************************************************************************.............**...****************..********X | 109.700 | | 90.300 | ..............****X | 90.300 | | 80.100 | ............X | 80.100 | | 50.000 | .......*******X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ********************************************************************************************************************************************....******.........................................*******X | 270.900 | | 29.100 | ...*X | 29.100 | | 0.000 | ******************************************************************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 14:58:42,632 | test | INFO | pool-11-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..******................................................................................***..................****X | 64742.300 | | 45278.900 | ....................................................................**X | 45278.900 | | 41694.200 | ...............................................................******...........**...******..................**..*****X | 63475.900 | | 32934.900 | .................................................*X | 32934.900 | | 24040.900 | ....................................*****X | 24040.900 | | 17082.500 | .........................**X | 17082.500 | | 16844.300 | .........................****X | 16844.300 | | 11856.500 | .................**X | 11856.500 | | 11673.000 | .................*****..................................***...........*****............................***....****X | 61937.000 | | 46085.900 | ......................................................................**X | 46085.900 | | 40206.700 | .............................................................****X | 40206.700 | | 37815.500 | .........................................................***...*****................***....****X | 52533.200 | | 42651.300 | ................................................................**X | 42651.300 | | 38853.300 | ...........................................................****X | 38853.300 | | 28281.000 | ..........................................**X | 28281.000 | | 26826.300 | ........................................****X | 26826.300 | | 23189.700 | ..................................**X | 23189.700 | | 21016.700 | ...............................****X | 21016.700 | | 13295.200 | ...................**X | 13295.200 | | 12441.100 | ..................******...**.******....**.******.............**...*****X | 28960.300 | | 23815.500 | ...................................*X | 23815.500 | | 20972.200 | ...............................*****X | 20972.200 | | 17046.400 | .........................********......********...***...*****...........**X | 31767.000 | | 31679.400 | ...............................................****X | 31679.400 | | 24075.200 | ....................................**X | 24075.200 | | 22653.900 | .................................*****X | 22653.900 | | 19539.200 | .............................*X | 19539.200 | | 19003.000 | ............................*****X | 19003.000 | | 18961.100 | ............................*******X | 19211.700 | | 18511.900 | ...........................*X | 18511.900 | | 17481.200 | ..........................*****X | 17481.200 | | 12074.500 | .................*X | 12074.500 | | 10936.200 | ...............******...***..*****...********......***...*****..**X | 22924.100 | | 22427.200 | .................................*****X | 22427.200 | | 20771.200 | ...............................********..**.******........**.*****X | 28776.800 | | 22669.300 | ..................................*X | 22669.300 | | 21083.000 | ...............................*****X | 21083.000 | | 19184.100 | ............................*X | 19184.100 | | 18690.900 | ...........................*****X | 18690.900 | | 17676.300 | ..........................*******X | 17933.300 | | 14734.900 | .....................*X | 14734.900 | | 14131.100 | ....................******.....**X | 17288.500 | | 17152.000 | .........................****X | 17152.000 | | 16121.100 | .......................*********.**..******.....********....*******X | 23720.300 | | 19905.800 | .............................*X | 19905.800 | | 18976.800 | ............................*****X | 18976.800 | | 13194.400 | ...................*X | 13194.400 | | 11723.100 | .................**************.........***...*****....********...***.****X | 24673.200 | | 22240.200 | .................................**X | 22240.200 | | 21823.900 | ................................*****X | 21823.900 | | 17725.600 | ..........................*X | 17725.600 | | 16332.800 | ........................*****X | 16332.800 | | 13763.000 | ....................*******..********.*********..........**...*****X | 24266.000 | | 22207.500 | .................................*X | 22207.500 | | 20674.100 | ..............................*****X | 20674.100 | | 19076.500 | ............................********..**..******........*X | 27043.800 | | 25899.700 | .......................................*****X | 25899.700 | | 17281.300 | .........................*X | 17281.300 | | 16596.300 | ........................******.....***.****X | 20259.000 | | 17682.600 | ..........................**X | 17682.600 | | 17534.300 | ..........................*****..............***..****X | 27952.700 | | 20246.700 | ..............................**X | 20246.700 | | 18782.700 | ............................******.*******X | 19863.200 | | 17137.700 | .........................*X | 17137.700 | | 16388.600 | ........................******.......**...*****X | 22853.400 | | 22446.900 | .................................*X | 22446.900 | | 21875.400 | ................................*****X | 21875.400 | | 14630.100 | .....................**X | 14630.100 | | 12818.700 | ..................*****..***.*****X | 14366.600 | | 13947.600 | ....................*X | 13947.600 | | 13824.800 | ....................******..********....**..******..**X | 20232.100 | | 19579.500 | .............................****X | 19579.500 | | 17101.800 | .........................**X | 17101.800 | | 16699.400 | ........................******...**...******........*X | 25818.400 | | 25107.800 | .....................................*****X | 25107.800 | | 16532.700 | ........................*X | 16532.700 | | 15731.500 | .......................******..**.******..**X | 19317.700 | | 19097.000 | ............................****X | 19097.000 | | 15413.600 | ......................**X | 15413.600 | | 15184.000 | ......................******..*X | 16516.100 | | 16461.200 | ........................********.**************....**.******...*******X | 22552.900 | | 18504.500 | ...........................**X | 18504.500 | | 17204.600 | .........................****X | 17204.600 | | 14270.500 | .....................**X | 14270.500 | | 13889.200 | ....................******........**...*****X | 21002.600 | | 20040.600 | .............................*X | 20040.600 | | 19385.000 | ............................******...**.*****X | 21761.500 | | 19385.500 | ............................*X | 19385.500 | | 18687.200 | ...........................******.**X | 18794.600 | | 18489.300 | ...........................*****...***..*****X | 21433.900 | | 19787.900 | .............................*X | 19787.900 | | 18928.000 | ............................*****X | 18928.000 | | 14709.000 | .....................*X | 14709.000 | | 13502.800 | ...................******...**..******...********.*******X | 19185.700 | | 16730.500 | ........................**X | 16730.500 | | 15725.500 | .......................******....**.*****X | 19413.200 | | 18396.700 | ...........................*X | 18396.700 | | 17701.900 | ..........................******.**.******........*******X | 24432.500 | | 16370.900 | ........................*X | 16370.900 | | 15249.600 | ......................******...**..*****X | 18136.000 | | 17029.100 | .........................**X | 17029.100 | | 15977.000 | .......................****X | 15977.000 | | 15897.700 | .......................********.***.*****......**X | 21015.900 | | 20723.800 | ...............................*****X | 20723.800 | | 18000.000 | ..........................**.******..*X | 19516.200 | | 19361.100 | ............................*****X | 19361.100 | | 17402.300 | .........................*X | 17402.300 | | 16894.500 | .........................*****X | 16894.500 | | 16303.500 | ........................**X | 16303.500 | | 16223.200 | ........................*****.**X | 17343.400 | | 17308.700 | .........................********.******.......**.*****X | 22867.600 | | 21528.600 | ................................*X | 21528.600 | | 21256.100 | ...............................******.*X | 21529.700 | | 20975.000 | ...............................*****X | 20975.000 | | 15956.600 | .......................*X | 15956.600 | | 15094.700 | ......................******....**..*****X | 18857.900 | | 17878.800 | ..........................********....***.****X | 21021.200 | | 18133.700 | ...........................**X | 18133.700 | | 16899.500 | .........................*****.......***.*****X | 22497.600 | | 13725.500 | ....................*X | 13725.500 | | 13039.800 | ...................******......**.******...*******X | 19908.100 | | 17554.800 | ..........................*X | 17554.800 | | 17297.000 | .........................******..**.*****X | 19160.700 | | 17533.900 | ..........................*X | 17533.900 | | 16611.500 | ........................*****X | 16611.500 | | 14007.500 | ....................********........*********..**.*****X | 21108.600 | | 15385.400 | ......................*X | 15385.400 | | 14271.900 | .....................******....********....*X | 19721.500 | | 19719.000 | .............................*****X | 19719.000 | | 13796.400 | ....................*X | 13796.400 | | 13538.900 | ...................******...*X | 15247.400 | | 14969.800 | ......................******......***.****X | 19837.700 | | 17749.100 | ..........................*******X | 17864.900 | | 17396.100 | .........................**X | 17396.100 | | 17049.800 | .........................******...*X | 19049.500 | | 18565.700 | ...........................*****X | 18565.700 | | 15198.400 | ......................**.*******X | 15705.500 | | 15480.900 | ......................******...........**.*****X | 23224.700 | | 19961.600 | .............................*X | 19961.600 | | 19672.600 | .............................*****X | 19672.600 | | 16195.000 | ........................*X | 16195.000 | | 15994.800 | .......................******........***..****X | 22031.600 | | 19768.100 | .............................**X | 19768.100 | | 19280.800 | ............................*****X | 19280.800 | | 15888.300 | .......................*X | 15888.300 | | 15464.600 | ......................******...*******X | 17233.900 | | 16683.800 | ........................********.....**.*****X | 20108.300 | | 16094.700 | .......................*X | 16094.700 | | 16076.900 | .......................****X | 16076.900 | +-----------+-------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 14:58:45,124 | test | INFO | pool-11-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 16076.900 | ...................................................................................................X | 16076.900 | | 15072.500 | ............................................................................................X | 15072.500 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 14:58:45,232 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 42.5409416581 | 11.45 GiB | 8.45 GiB | 52.50 MiB / 3.50 GiB | 6471750 / 13069483 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 30.6252387623 | 11.45 GiB | 8.77 GiB | 24.00 MiB / 3.50 GiB | 6543750 / 13049100 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 46.0451977401 | 11.45 GiB | 8.45 GiB | 28.14 MiB / 3.50 GiB | 6524445 / 12986779 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 46.6006939982 | 11.45 GiB | 8.50 GiB | 0.0 Byte / 3.50 GiB | 6519861 / 13052483 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 25.4144350931 | 11.45 GiB | 8.68 GiB | 0.0 Byte / 3.50 GiB | 6510202 / 12971193 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 14:58:53,443 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 14:59:07,663 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 14:59:07,664 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 14:59:18,480 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 14:59:18,480 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 14:59:29,444 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 14:59:29,444 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 14:59:47,766 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 246.95 MiB | 338.02 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 513.42 MiB | 574.74 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.77 GiB | 51.67 GiB | 20.0999017652 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 14:59:47,766 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 14:59:52,088 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during hard_failover_recovery 2022-02-02 14:59:52,088 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : hard_failover_recovery 2022-02-02 14:59:52,089 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:755] failing over nodes [ip:172.23.105.36 port:8091 ssh_username:root] 2022-02-02 15:00:01,233 | test | INFO | pool-11-thread-20 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03500008583 seconds to complete 2022-02-02 15:00:01,236 | test | INFO | pool-11-thread-20 | [common_lib:sleep:22] Sleep 8.03500008583 seconds. Reason: Wait after rebalance complete 2022-02-02 15:05:09,273 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 15:05:09,275 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 15:05:09,280 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 15:16:10,194 | test | INFO | pool-11-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******...............................................................................................*******X | 152.000 | | 42.000 | .........................********........................*******X | 80.000 | | 32.000 | ...................*********................................................................................*******X | 160.000 | | 64.000 | .......................................********................*******X | 90.000 | | 40.000 | ........................********.....................................*******X | 100.000 | | 92.000 | ........................................................*******X | 92.000 | | 36.200 | .....................X | 36.200 | | 18.000 | ..........*******X | 18.000 | | 0.000 | ********...................................................................*******X | 108.000 | | 32.000 | ...................********................................................*******X | 110.000 | | 58.000 | ...................................********................................*******X | 110.000 | | 34.000 | ....................********.....................................................................*******X | 144.000 | | 24.000 | ..............********...............................................*******X | 100.000 | | 48.000 | .............................*********.........********....*******X | 69.200 | | 62.000 | .....................................*******X | 62.000 | | 32.000 | ...................*******X | 32.000 | | 28.800 | .................X | 28.800 | | 18.000 | ..........********........................................******X | 82.000 | | 0.000 | ********.....................................*******X | 60.000 | | 48.000 | .............................*******X | 48.000 | | 30.000 | .................********.......................********.....................*******X | 100.000 | | 0.000 | ********...................................................***************X | 82.000 | | 26.000 | ...............********........................*******X | 64.000 | | 48.000 | .............................*******X | 48.000 | | 38.000 | ......................*******X | 38.000 | | 26.000 | ...............********............*******X | 46.000 | | 8.000 | ....*********.................********.....*******X | 44.000 | | 24.000 | ..............*******X | 24.000 | | 10.000 | .....********...........................********...........................*******X | 96.000 | | 74.000 | .............................................********X | 74.000 | | 0.000 | ********.....................................*******X | 60.000 | | 32.000 | ...................*******X | 32.000 | | 18.000 | ..........********.............................*******X | 64.000 | | 8.000 | ....********...............********....................*******X | 64.000 | | 8.000 | ....********..........*******X | 24.000 | | 16.000 | .........*****************************X | 16.000 | +--------+----------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 15:16:10,645 | test | INFO | pool-11-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******....................................................................................................*******X | 380.400 | | 65.600 | ................********.................*******X | 130.000 | | 76.000 | ..................********..............*******X | 128.000 | | 124.000 | ...............................*******X | 124.000 | | 32.000 | .......********X | 32.000 | | 24.000 | .....********.................*******X | 90.000 | | 58.000 | ..............******X | 58.000 | | 58.000 | ..............*..............*******X | 112.000 | | 8.000 | .********....********.......********..********.********...********..*.********.*******X | 88.000 | | 56.000 | .............********.......*******X | 82.000 | | 74.000 | ..................*******X | 74.000 | | 72.000 | .................*******X | 72.000 | | 28.000 | ......********........***X | 60.000 | | 60.000 | ..............***X | 60.000 | | 60.000 | ..............X | 60.000 | | 38.000 | ........*******X | 38.000 | | 36.000 | ........********.........******X | 70.000 | | 46.000 | ...........*******X | 46.000 | | 28.000 | ......********..********......*******X | 60.000 | | 46.000 | ...........*******X | 46.000 | | 18.000 | ...*******X | 18.000 | | 16.000 | ...********..............********.....*******X | 88.000 | | 52.000 | ............*******X | 52.000 | | 8.000 | .********..........********.......*******X | 74.000 | | 26.000 | .....********......*******X | 46.000 | | 24.000 | .....********..*******X | 34.000 | | 26.000 | .....********X | 26.000 | | 8.000 | .********......********.......********...........*******X | 100.000 | | 32.000 | .......*******X | 32.000 | | 24.000 | .....****************X | 26.000 | | 8.000 | .********....***************X | 24.000 | | 10.000 | .*******X | 10.000 | | 8.000 | .************************..*******X | 16.000 | | 8.000 | .********....*********************X | 24.000 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 15:16:10,739 | test | INFO | pool-11-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******....................................................********............********...********..********................*******X | 45436.800 | | 39842.600 | ..........................................................................*******X | 39842.600 | | 36038.200 | ...................................................................********....********.........................*******X | 51791.800 | | 37099.700 | .....................................................................********............********..*******X | 44822.200 | | 42544.100 | ...............................................................................*******X | 42544.100 | | 41374.900 | .............................................................................****************......********..............*******X | 52172.900 | | 40269.600 | ...........................................................................*X | 40269.600 | | 40269.600 | ...........................................................................*****X | 40269.600 | | 36913.600 | ....................................................................X | 36913.600 | | 36913.600 | ....................................................................****X | 36913.600 | | 36913.600 | ....................................................................**...............................X | 52963.900 | | 52963.900 | ..................................................................................................*****.*X | 52963.900 | | 41665.000 | .............................................................................*******X | 41665.000 | | 31257.700 | ..........................................................********....*******X | 33719.700 | | 31109.600 | .........................................................*********......********.*******X | 34849.300 | | 31220.800 | .........................................................********.........*******.*******X | 36186.300 | | 33620.500 | ..............................................................********.********.......*******X | 37814.000 | | 35447.100 | .................................................................*******X | 35447.100 | | 35244.300 | .................................................................********..*******..*.******X | 38089.000 | | 35469.400 | .................................................................*********.............*******X | 42084.300 | | 26173.700 | ................................................********........********...*******X | 31933.100 | | 24624.800 | .............................................********.....********....*******X | 29498.100 | | 27151.000 | ..................................................********...********.........*******X | 33804.400 | | 31682.700 | ..........................................................********X | 31682.700 | | 26298.900 | ................................................********....................*******X | 36971.900 | | 29964.800 | .......................................................*******X | 29964.800 | | 29424.300 | ......................................................********....*******X | 31324.600 | | 23076.900 | ..........................................********............*******X | 29537.100 | | 27405.800 | ..................................................********.....*******X | 29988.500 | | 28061.000 | ...................................................*********....*******X | 30132.700 | | 27807.000 | ...................................................*******X | 27807.000 | | 26473.100 | ................................................********.....*******X | 28910.200 | | 22798.000 | ..........................................********..********..*****X | 24922.700 | +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 15:16:11,986 | test | INFO | pool-11-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 16.000 | ...................................................................................................X | 16.000 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 15:16:12,000 | test | INFO | pool-11-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 24.000 | ...................................................................................................X | 24.000 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 15:16:12,019 | test | INFO | pool-11-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 24922.700 | ...................................................................................................X | 24922.700 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 15:16:12,130 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 15:16:12,154 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.105.36 - Node not part of cluster inactiveFailed 2022-02-02 15:16:22,362 | test | INFO | pool-11-thread-12 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 0.4636010525 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 28.1817028296 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 38.5428607821 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 36.954015627 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 34.426646326 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 15:16:22,381 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 15:16:32,407 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 15:16:42,450 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 49.65 2022-02-02 15:16:52,487 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:17:02,525 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:17:12,558 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:17:22,592 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:17:32,628 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:17:42,674 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:17:52,710 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:18:02,743 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:18:12,782 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.28 2022-02-02 15:18:22,816 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 60.47 2022-02-02 15:18:32,851 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 65.12 2022-02-02 15:18:42,885 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: running, progress: 71.4 2022-02-02 15:18:57,934 | test | INFO | pool-11-thread-12 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 15:18:57,954 | test | INFO | pool-11-thread-12 | [task:check:468] Rebalance completed with progress: 100% in 155.592000008 sec 2022-02-02 15:18:57,954 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:755] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 15:19:07,124 | test | INFO | pool-11-thread-1 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03700017929 seconds to complete 2022-02-02 15:19:07,125 | test | INFO | pool-11-thread-1 | [common_lib:sleep:22] Sleep 8.03700017929 seconds. Reason: Wait after rebalance complete 2022-02-02 15:24:15,164 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 15:24:15,165 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 15:24:15,171 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 15:38:01,109 | test | INFO | pool-11-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **................................................................**....................................*****X | 273.600 | | 123.200 | ............................................*X | 123.200 | | 40.000 | .............******.......***.......****X | 79.000 | | 49.600 | .................X | 49.600 | | 49.600 | .................X | 49.600 | | 16.200 | ....******..................**.............*****X | 100.800 | | 55.200 | ...................*X | 55.200 | | 35.800 | ............******.....*X | 50.700 | | 47.800 | ................******..............**..........*****X | 114.000 | | 65.000 | ......................*X | 65.000 | | 54.200 | ..................******...............*X | 94.900 | | 87.300 | ..............................X | 87.300 | | 87.300 | ..............................****X | 87.300 | | 67.700 | .......................**..******.**X | 75.100 | | 66.500 | .......................****X | 66.500 | | 30.700 | ..........**X | 30.700 | | 25.300 | ........******..................**.....*****X | 87.800 | | 37.500 | ............*X | 37.500 | | 20.400 | ......******....................**.......**X | 94.700 | | 94.700 | .................................**X | 94.700 | | 55.500 | ...................*X | 55.500 | | 32.600 | ..........******.......**.........*****X | 75.400 | | 52.800 | ..................**X | 52.800 | | 26.500 | ........*****.***..****X | 35.300 | | 27.000 | ........***....*****............***....******..*X | 85.000 | | 72.700 | .........................******......**.....*****X | 102.100 | | 61.200 | .....................*X | 61.200 | | 39.500 | .............*****X | 39.500 | | 37.800 | ............*X | 37.800 | | 37.300 | ............*****X | 37.300 | | 17.700 | .....**X | 17.700 | | 9.600 | ..*****...............***.........****X | 76.000 | | 71.800 | .........................X | 71.800 | | 60.200 | .....................*X | 60.200 | | 42.400 | ..............****X | 42.400 | | 33.700 | ...........**********X | 34.600 | | 34.400 | ...........*****X | 34.400 | | 27.800 | .........**...******.....*X | 51.400 | | 45.500 | ...............******.**X | 47.400 | | 43.000 | ..............****X | 43.000 | | 36.800 | ............***..*****X | 41.400 | | 38.000 | ............*X | 38.000 | | 28.000 | .........*****X | 28.000 | | 24.200 | .......**....******....*X | 44.500 | | 41.000 | .............*****X | 41.000 | | 21.800 | ......**X | 21.800 | | 18.200 | .....*****.......***..*****X | 41.500 | | 30.000 | .........*X | 30.000 | | 23.600 | .......******.......**....*****X | 53.600 | | 29.500 | .........*X | 29.500 | | 17.200 | .....********.********X | 20.400 | | 17.200 | .....*****....***..******.........**......*****X | 76.400 | | 41.900 | ..............*X | 41.900 | | 24.800 | ........********..******..*X | 36.900 | | 28.700 | .........*****X | 28.700 | | 5.700 | .**X | 5.700 | | 3.200 | ******...........**....*****X | 45.300 | | 36.300 | ............*X | 36.300 | | 29.900 | .........*****X | 29.900 | | 14.600 | ....*X | 14.600 | | 9.600 | ..******..**....******.....**X | 38.900 | | 27.500 | .........****X | 27.500 | | 14.700 | ....***.******..**.*****X | 24.800 | | 13.300 | ...*X | 13.300 | | 7.600 | .******.***.*****......***.....*****....**X | 53.900 | | 43.200 | ..............*****X | 43.200 | | 20.300 | ......*******X | 21.700 | | 19.100 | .....*X | 19.100 | | 12.700 | ...********.*****X | 15.200 | | 5.000 | *X | 5.000 | | 0.000 | ******.***.*****..***.....******.........*X | 50.700 | | 43.200 | ..............*****X | 43.200 | | 12.600 | ...*X | 12.600 | | 10.800 | ..******.........**...*****X | 41.300 | | 33.600 | ...........**X | 33.600 | | 27.900 | .........****X | 27.900 | | 22.800 | .......********X | 22.800 | | 13.900 | ....*X | 13.900 | | 9.600 | ..******...**..*****X | 22.800 | | 12.600 | ...**X | 12.600 | | 7.600 | .*****......***....****X | 32.900 | | 22.700 | .......**X | 22.700 | | 12.700 | ...****X | 12.700 | | 7.600 | .***.******..*X | 15.200 | | 12.700 | ...*****X | 12.700 | | 7.600 | .*******X | 7.600 | | 2.500 | *X | 2.500 | | 0.000 | ***X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 15:38:01,625 | test | INFO | pool-11-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **................................................................**....................................*****X | 154.000 | | 65.800 | .........................................*X | 65.800 | | 17.200 | ..........******.....**.......*****X | 35.800 | | 26.000 | ...............*X | 26.000 | | 9.800 | .....******...........................................**................................*****X | 126.100 | | 88.800 | ........................................................*X | 88.800 | | 63.500 | ........................................*******X | 63.500 | | 50.000 | ...............................******...**.........*****X | 68.400 | | 43.900 | ...........................**..******....................*X | 77.500 | | 62.000 | .......................................*****X | 62.000 | | 50.500 | ...............................**.......*****X | 60.800 | | 50.800 | ...............................*X | 50.800 | | 44.000 | ...........................*****X | 44.000 | | 19.700 | ...........**.******......................**X | 55.400 | | 50.000 | ...............................****X | 50.000 | | 40.400 | .........................***..........******...*X | 61.100 | | 39.700 | ........................*****X | 39.700 | | 20.100 | ............**...******................................................**.........................*****X | 137.900 | | 84.200 | .....................................................*X | 84.200 | | 47.800 | ..............................*****X | 47.800 | | 23.300 | ..............*X | 23.300 | | 15.200 | ........******..............................***..................***X | 88.800 | | 88.800 | ........................................................X | 88.800 | | 69.200 | ...........................................**X | 69.200 | | 62.600 | .......................................****X | 62.600 | | 61.700 | .......................................**X | 61.700 | | 53.900 | ..................................****X | 53.900 | | 51.400 | ................................**X | 51.400 | | 46.700 | .............................******....*X | 53.300 | | 50.800 | ...............................*****X | 50.800 | | 32.900 | ....................*X | 32.900 | | 32.400 | ....................******.................**.............*****X | 79.800 | | 41.000 | .........................**X | 41.000 | | 9.800 | .....*****...***.....*****.......***...****X | 38.000 | | 31.200 | ...................**X | 31.200 | | 29.500 | ..................****X | 29.500 | | 26.700 | ................***.******....................**......*****X | 68.300 | | 61.700 | .......................................*X | 61.700 | | 56.700 | ...................................*****X | 56.700 | | 40.300 | .........................*X | 40.300 | | 28.500 | .................*****X | 28.500 | | 23.600 | ..............**..*****X | 26.800 | | 20.400 | ............*X | 20.400 | | 17.200 | ..........******...........***........****X | 47.200 | | 31.200 | ...................X | 31.200 | | 24.600 | ..............*X | 24.600 | | 9.100 | ....******...............**..........******.........*X | 60.100 | | 59.600 | .....................................*****X | 59.600 | | 28.100 | .................**X | 28.100 | | 26.100 | ...............*****.....................**X | 57.000 | | 54.300 | ..................................*****X | 54.300 | | 18.500 | ...........*X | 18.500 | | 9.600 | .....******.......**....*****X | 26.800 | | 20.600 | ............*X | 20.600 | | 17.200 | ..........******..................**.........*****X | 59.200 | | 19.600 | ...........**X | 19.600 | | 0.000 | *****........................***.............*****X | 57.600 | | 19.200 | ...........*X | 19.200 | | 0.000 | ******.......**.......******......*X | 31.100 | | 25.200 | ...............*****X | 25.200 | | 17.800 | ..........**.*****X | 19.200 | | 6.400 | ...*********........*X | 19.200 | | 16.000 | .........******.......**....*****X | 33.200 | | 21.700 | .............*X | 21.700 | | 19.200 | ...........*********..*****...**X | 26.800 | | 24.300 | ..............*****X | 24.300 | | 17.900 | ..........**........******.....................*X | 62.300 | | 59.800 | .....................................*****X | 59.800 | | 56.000 | ...................................**X | 56.000 | | 50.800 | ...............................****X | 50.800 | | 11.400 | ......***......******......................................*X | 79.100 | | 70.900 | .............................................*****X | 70.900 | | 46.600 | .............................**...*****X | 50.900 | | 48.300 | ..............................**.*****X | 50.100 | | 28.400 | .................*X | 28.400 | | 10.200 | .....******.............**............******....**X | 55.300 | | 48.200 | ..............................****X | 48.200 | | 12.500 | .......**X | 12.500 | | 2.500 | *****......*............**........*****X | 42.500 | | 31.000 | ...................*X | 31.000 | | 17.800 | ..........********...******..........**.....*****X | 45.600 | | 36.700 | ......................**X | 36.700 | | 32.400 | ....................****X | 32.400 | | 15.800 | .........**X | 15.800 | | 7.600 | ...******....**.*****....***.******.......**............******.*X | 52.500 | | 29.300 | ..................*****X | 29.300 | | 17.800 | ..........***.....****X | 25.300 | | 20.200 | ............**X | 20.200 | | 10.200 | .....***X | 10.200 | +--------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 15:38:02,351 | test | INFO | pool-11-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...**....******..................**......******................................**...............*****X | 41375.300 | | 34788.900 | ................................................................*X | 34788.900 | | 30446.800 | ........................................................*****X | 30446.800 | | 30326.800 | ........................................................**...******..*X | 33264.100 | | 31968.300 | ...........................................................*****X | 31968.300 | | 31190.300 | .........................................................**....******...........*X | 39064.100 | | 38561.700 | .......................................................................*****X | 38561.700 | | 37204.600 | .....................................................................**..******.....**.....******..*X | 44507.000 | | 43247.200 | ................................................................................*****X | 43247.200 | | 38069.900 | ......................................................................*X | 38069.900 | | 35034.700 | .................................................................*****X | 35034.700 | | 32814.500 | .............................................................********....**.***X | 35907.100 | | 35907.100 | ..................................................................*X | 35907.100 | | 35013.000 | .................................................................*X | 35013.000 | | 34419.400 | ................................................................*****X | 34419.400 | | 32427.200 | ............................................................**X | 32427.200 | | 31160.200 | .........................................................*****.....***.....*****......................***..........****X | 52894.300 | | 51351.100 | ................................................................................................**X | 51796.400 | | 46263.500 | ......................................................................................*****X | 46263.500 | | 40950.400 | ............................................................................*X | 40950.400 | | 39627.700 | .........................................................................*****X | 39627.700 | | 34709.400 | ................................................................*******X | 34894.200 | | 30323.800 | ........................................................*X | 30323.800 | | 28725.000 | .....................................................******......**.*****X | 32444.900 | | 31320.500 | ..........................................................**...*****X | 32833.600 | | 31512.300 | ..........................................................**X | 31512.300 | | 30317.200 | ........................................................*****.......*******X | 34297.400 | | 28090.700 | ....................................................**X | 28090.700 | | 27232.800 | ..................................................*****X | 27232.800 | | 26813.900 | .................................................*X | 26813.900 | | 26150.500 | ................................................*****.......***....*****................X | 40722.100 | | 40722.100 | ...........................................................................X | 40722.100 | | 40722.100 | ...........................................................................*.......****X | 44021.300 | | 34560.700 | ................................................................**X | 34560.700 | | 28653.400 | .....................................................*****.....***....*******X | 33638.200 | | 33247.900 | .............................................................*****X | 33247.900 | | 30107.300 | .......................................................*X | 30107.300 | | 29135.100 | ......................................................*****X | 29135.100 | | 21959.200 | ........................................*X | 21959.200 | | 18129.600 | .................................******...........**...........*****X | 29778.000 | | 29586.900 | ......................................................*X | 29586.900 | | 27055.800 | ..................................................*****X | 27055.800 | | 23337.400 | ...........................................**X | 23337.400 | | 22609.900 | .........................................********..****X | 23413.900 | | 23014.100 | ..........................................**X | 23014.100 | | 21933.700 | ........................................****X | 21933.700 | | 19609.900 | ....................................**X | 19609.900 | | 19054.100 | ...................................*****X | 19054.100 | | 18899.500 | ..................................**.*****X | 19409.800 | | 18751.900 | ..................................*X | 18751.900 | | 17684.200 | ................................******..**..******....**..*****X | 22769.500 | | 22148.800 | ........................................*X | 22148.800 | | 21615.800 | .......................................******.....**..*****X | 25212.600 | | 23376.200 | ...........................................**X | 23376.200 | | 21950.200 | ........................................****X | 21950.200 | | 21119.900 | ......................................**X | 21119.900 | | 20604.300 | .....................................****X | 20604.300 | | 20211.300 | .....................................*********.*******X | 20818.700 | | 19340.500 | ...................................*X | 19340.500 | | 18608.700 | ..................................******.....**..*****X | 22476.200 | | 19773.700 | ....................................**X | 19773.700 | | 19132.700 | ...................................*****.**X | 19585.400 | | 19307.300 | ...................................*****...***..******.*X | 22314.100 | | 22218.400 | .........................................*****X | 22218.400 | | 19341.900 | ...................................*X | 19341.900 | | 18536.100 | ..................................******......**....*****X | 24062.300 | | 22444.600 | .........................................**X | 22444.600 | | 20875.900 | ......................................****X | 20875.900 | | 17393.900 | ...............................**X | 17393.900 | | 16800.500 | ..............................******...........**....*****X | 24695.100 | | 22951.100 | ..........................................*X | 22951.100 | | 21337.700 | .......................................******.**.*****X | 22619.000 | | 21131.600 | ......................................*X | 21131.600 | | 20909.800 | ......................................******...*******X | 22589.900 | | 18327.100 | .................................*X | 18327.100 | | 17293.300 | ...............................******...............**......*****X | 28081.400 | | 21842.500 | ........................................**X | 21842.500 | | 19878.600 | ....................................*****...**X | 21437.600 | | 21104.900 | ......................................*****..***.*****X | 22692.500 | | 20837.200 | ......................................*X | 20837.200 | | 19209.000 | ...................................*****X | 19209.000 | | 19094.100 | ...................................**.******.*X | 20324.000 | | 19763.200 | ....................................*****X | 19763.200 | | 17822.200 | ................................*******X | 17946.400 | | 16213.700 | .............................**X | 16213.700 | | 15301.300 | ...........................******......**...****X | 19715.600 | | 19002.300 | ..................................**X | 19002.300 | | 18225.600 | .................................****X | 18225.600 | | 18095.900 | .................................********...********X | 19942.600 | | 17640.200 | ................................*X | 17640.200 | | 17210.400 | ...............................*******X | 17312.600 | | 17235.100 | ...............................****X | 17235.100 | +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 15:38:03,619 | test | INFO | pool-11-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 10.200 | ...................................................................................................X | 10.200 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 15:38:03,645 | test | INFO | pool-11-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 17235.100 | ...................................................................................................X | 17235.100 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 15:38:03,756 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 15:38:03,778 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 15:38:11,440 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 15:38:11,447 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 15:38:11,677 | test | INFO | pool-11-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 16.9696969697 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 24.9302207562 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 31.546785487 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.525722868945 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 26.1051829268 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 15:38:11,694 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 15:38:21,723 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 15:38:31,772 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 51.94 2022-02-02 15:38:41,812 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:38:51,858 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:39:01,900 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:39:11,940 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:39:21,977 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:39:32,013 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:39:42,053 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:39:52,095 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:40:02,134 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:40:12,176 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:40:22,213 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:40:32,250 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 60.0 2022-02-02 15:40:42,292 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 62.75 2022-02-02 15:40:52,336 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 66.36 2022-02-02 15:41:02,380 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:41:12,415 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:41:22,453 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:41:32,496 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:41:42,540 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:41:52,581 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:42:02,621 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:42:12,661 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:42:22,696 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:42:32,734 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:42:42,779 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:42:52,819 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 15:43:02,868 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.25 2022-02-02 15:43:12,903 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.31 2022-02-02 15:43:22,950 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.31 2022-02-02 15:43:32,990 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.6 2022-02-02 15:43:43,042 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 67.98 2022-02-02 15:43:53,085 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 71.59 2022-02-02 15:44:03,132 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 76.08 2022-02-02 15:44:13,174 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 77.16 2022-02-02 15:44:23,213 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 77.85 2022-02-02 15:44:33,253 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 83.85 2022-02-02 15:44:43,293 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: running, progress: 86.56 2022-02-02 15:44:58,357 | test | INFO | pool-11-thread-30 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 15:44:58,380 | test | INFO | pool-11-thread-30 | [task:check:468] Rebalance completed with progress: 100% in 406.703000069 sec 2022-02-02 15:52:29,903 | test | INFO | pool-11-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****...................................................................**..............*****X | 111.400 | | 62.800 | .............................................*X | 62.800 | | 51.500 | ....................................*****X | 51.500 | | 8.400 | .....*X | 8.400 | | 0.000 | ******........**..******.....**..*****X | 24.300 | | 18.200 | ............**X | 18.200 | | 14.600 | .........*****..................***..*****X | 41.400 | | 17.000 | ...........*X | 17.000 | | 15.800 | ..........******......................................................**.........*****X | 101.000 | | 15.600 | ..........*X | 15.600 | | 0.000 | ******......................***....*****..................***....*****X | 65.700 | | 59.600 | ..........................................*X | 59.600 | | 58.400 | .........................................*****X | 58.400 | | 15.700 | ..........*X | 15.700 | | 7.300 | ....********.******.....................................**......*****X | 67.600 | | 17.500 | ...........*X | 17.500 | | 6.100 | ...******..............**......*****X | 33.500 | | 32.200 | ......................*X | 32.200 | | 29.200 | ....................******........................................**.......*****X | 92.500 | | 37.600 | ..........................*X | 37.600 | | 29.200 | ....................*****X | 29.200 | | 23.100 | ...............**..******.*X | 26.200 | | 22.000 | ...............******.....................................................................**...............*****X | 135.900 | | 34.400 | ........................*X | 34.400 | | 15.800 | ..........********..*****...............................*******X | 60.900 | | 18.200 | ............**X | 18.200 | | 14.600 | .........********..******....................................**..........******......*X | 88.300 | | 76.900 | .......................................................*X | 76.900 | | 76.900 | .......................................................***X | 76.900 | | 25.500 | .................*X | 25.500 | | 23.100 | ...............******............................................**.......*****X | 92.100 | | 15.000 | ..........*X | 15.000 | | 0.000 | ******.....***.........*****..............................................*X | 81.700 | | 73.900 | .....................................................*****X | 73.900 | | 22.500 | ...............*X | 22.500 | | 21.900 | ...............******.....................***.......****X | 59.900 | | 57.300 | .........................................**X | 57.300 | | 53.700 | ......................................*****X | 53.700 | | 41.500 | .............................*******X | 41.500 | | 28.000 | ...................*X | 28.000 | | 26.800 | ..................******...................................**...........******................*X | 111.300 | | 99.300 | ........................................................................*****X | 99.300 | | 9.600 | ......*X | 9.600 | | 0.000 | ******..................................***...............*****...............**X | 87.900 | | 74.100 | .....................................................****X | 74.100 | | 21.400 | ..............**X | 21.400 | | 17.800 | ............******..............**......******....*X | 51.300 | | 50.700 | ....................................*****X | 50.700 | | 32.400 | ......................*X | 32.400 | | 24.000 | ................******........**...*****X | 38.500 | | 5.400 | ..**X | 5.400 | | 3.000 | .*****..............................***..*****X | 46.500 | | 6.000 | ...*X | 6.000 | | 1.800 | ******.....................**...*****X | 34.900 | | 20.100 | .............*X | 20.100 | | 18.900 | ............******................**X | 39.800 | | 38.000 | ..........................****X | 38.000 | | 30.600 | .....................***..*****X | 33.000 | | 19.500 | .............*X | 19.500 | | 16.500 | ...........******.....**.*****X | 25.700 | | 4.800 | ..*X | 4.800 | | 3.000 | .******...............***....*****............................................***....*****X | 94.200 | | 19.300 | .............*X | 19.300 | | 7.300 | ....******.....**..******.......................**........*****X | 59.100 | | 49.500 | ...................................*X | 49.500 | | 41.100 | .............................*****X | 41.100 | | 23.700 | ................********................*X | 45.200 | | 44.600 | ...............................*****X | 44.600 | | 25.000 | .................**X | 25.000 | | 24.400 | ................*******X | 24.400 | | 21.400 | ..............*****X | 21.400 | | 7.900 | ....*X | 7.900 | | 7.300 | ....******.......................**.....*****X | 45.700 | | 13.900 | .........*X | 13.900 | | 7.300 | ....*****X | 7.300 | | 1.200 | *X | 1.200 | | 0.000 | ******......................***......****X | 38.900 | | 32.800 | .......................**X | 32.800 | | 26.800 | ..................******..........**....*****X | 45.700 | | 7.800 | ....*X | 7.800 | | 1.200 | ******..................**......******......**X | 40.800 | | 34.200 | ........................****X | 34.200 | | 32.900 | .......................***.******.........**...*****X | 51.100 | | 14.500 | .........*X | 14.500 | | 7.300 | ....*****X | 7.300 | | 1.200 | *X | 1.200 | | 0.000 | **X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 15:52:30,601 | test | INFO | pool-11-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *************......**.*****X | 7.600 | | 1.400 | *X | 1.400 | | 0.000 | ******...........***...*****X | 14.600 | | 2.400 | .********............*X | 14.600 | | 12.200 | ..........******..............................***........****X | 51.100 | | 20.600 | ..................**X | 20.600 | | 17.000 | ...............******........................**..*****X | 43.800 | | 31.600 | .............................*X | 31.600 | | 26.800 | ........................******............**.....*****X | 43.800 | | 31.600 | .............................**X | 31.600 | | 29.200 | ...........................*****......***...******......*X | 45.000 | | 42.600 | ........................................*****X | 42.600 | | 30.400 | ............................*X | 30.400 | | 28.000 | ..........................******...........**....*****X | 43.800 | | 25.500 | .......................*X | 25.500 | | 21.900 | ....................******......................................................**..............X | 91.900 | | 91.900 | ........................................................................................**X | 91.900 | | 91.900 | ........................................................................................*X | 91.900 | | 35.700 | .................................*X | 35.700 | | 20.700 | ...................******.................**....*****X | 42.600 | | 24.300 | ......................*X | 24.300 | | 21.900 | ....................******...............................**......*****X | 60.300 | | 22.400 | ....................*X | 22.400 | | 17.000 | ...............******..................................................................**...............*****X | 100.400 | | 77.300 | .........................................................................*X | 77.300 | | 67.100 | ................................................................****X | 67.100 | | 58.800 | ........................................................X | 58.800 | | 34.000 | ...............................*******..................**X | 52.300 | | 42.700 | ........................................****X | 42.700 | | 0.000 | ********...............................................***...........****X | 60.800 | | 51.200 | ................................................**X | 51.200 | | 46.400 | ............................................****X | 46.400 | | 44.400 | ..........................................X | 44.400 | | 37.700 | ...................................*X | 37.700 | | 35.300 | .................................*****....................***.........******.........................*X | 91.500 | | 89.700 | ......................................................................................X | 89.700 | | 89.700 | ......................................................................................*X | 89.700 | | 89.700 | ......................................................................................**X | 89.700 | | 26.100 | ........................*X | 26.100 | | 16.500 | ...............******.....**....******........................**.*****X | 51.800 | | 21.300 | ...................*X | 21.300 | | 15.300 | .............******.......................................................**............*****X | 84.100 | | 38.800 | ....................................*X | 38.800 | | 29.200 | ...........................******..........***......*****..................................................***.....****X | 102.600 | | 52.500 | .................................................**X | 52.500 | | 43.500 | .........................................******......................**....*****X | 70.200 | | 26.200 | ........................*X | 26.200 | | 18.400 | ................******..................**......*****X | 43.300 | | 29.800 | ...........................*X | 29.800 | | 24.400 | ......................******..................................**.......*****X | 66.000 | | 46.400 | ............................................**X | 46.400 | | 41.600 | .......................................****X | 41.600 | | 26.800 | ........................**X | 26.800 | | 25.000 | .......................******.......*X | 32.400 | | 31.800 | .............................******................................**.........*****X | 73.300 | | 26.700 | ........................*X | 26.700 | | 13.500 | ............******...............**.......*****X | 36.800 | | 14.600 | .............*X | 14.600 | | 9.200 | .......******........................**.....*****X | 38.400 | | 21.400 | ...................**X | 21.400 | | 20.200 | ..................*****................................***....*****X | 56.900 | | 22.500 | ....................*X | 22.500 | | 18.300 | ................******.........*X | 27.000 | | 25.800 | ........................******.................***......****X | 49.500 | | 9.000 | .......**X | 9.000 | | 0.000 | ********.******...................**.******..**.....******........*X | 38.000 | | 33.200 | ...............................*****X | 33.200 | | 30.600 | ............................***...****X | 33.000 | | 19.500 | .................***..******.....................................................................**...........*****X | 103.100 | | 63.900 | ............................................................*X | 63.900 | | 51.300 | ................................................*****X | 51.300 | | 39.100 | ....................................**..*****X | 40.900 | | 33.500 | ...............................*X | 33.500 | | 31.100 | .............................*****X | 31.100 | | 25.000 | .......................**X | 25.000 | | 21.400 | ...................*****.......***......****X | 34.200 | | 32.900 | ..............................**X | 32.900 | | 29.900 | ............................******.....**....******......**...******..............*X | 63.500 | | 59.900 | .........................................................*****X | 59.900 | | 43.800 | .........................................**.*****X | 45.000 | | 20.600 | ..................**X | 20.600 | | 15.800 | ..............*****............**X | 28.000 | | 26.800 | ........................*****......***....*****X | 36.500 | | 24.300 | ......................*X | 24.300 | | 23.100 | .....................******............**..*****X | 37.600 | | 31.500 | .............................*X | 31.500 | | 26.900 | .........................*****X | 26.900 | | 2.500 | .**X | 2.500 | | 0.000 | **X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 15:52:31,576 | test | INFO | pool-11-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****....................**..............******......................**...*****X | 24796.200 | | 16609.500 | ......................................*X | 16609.500 | | 14145.400 | ................................******..********....***....*****...**X | 19337.200 | | 18427.500 | ..........................................****X | 18427.500 | | 17022.900 | .......................................***.*****..............***.....****X | 25541.100 | | 22417.600 | ....................................................**X | 22417.600 | | 20715.200 | ................................................******.*X | 21135.200 | | 21045.700 | .................................................*****X | 21045.700 | | 19832.600 | ..............................................********..**..******.............**....*****X | 28864.300 | | 25783.000 | ............................................................**X | 25783.000 | | 23977.100 | ........................................................****X | 23977.100 | | 21256.800 | .................................................**X | 21256.800 | | 20069.300 | ..............................................******........**.....*****X | 25535.400 | | 23496.400 | .......................................................*X | 23496.400 | | 21879.300 | ...................................................*****X | 21879.300 | | 20020.000 | ..............................................*X | 20020.000 | | 19794.700 | ..............................................******...............**........******.................**.....****X | 38616.400 | | 38616.400 | ...........................................................................................X | 38616.400 | | 28370.400 | ..................................................................*X | 28370.400 | | 24453.100 | .........................................................*X | 24453.100 | | 24453.100 | .........................................................****....**.....******..********...*X | 30252.000 | | 30163.000 | ......................................................................*****X | 30163.000 | | 27252.400 | ................................................................*X | 27252.400 | | 26416.600 | ..............................................................*****............***......****X | 33942.100 | | 30237.900 | .......................................................................**X | 30237.900 | | 26753.100 | ..............................................................****X | 26753.100 | | 21453.000 | ..................................................**X | 21453.000 | | 21090.700 | .................................................*****..........***.....*****........................***......****X | 40072.700 | | 36380.500 | .....................................................................................**X | 36380.500 | | 35741.800 | ....................................................................................*****....***...****X | 38866.700 | | 37394.400 | ........................................................................................X | 37394.400 | | 37394.400 | ........................................................................................*X | 37394.400 | | 34060.600 | ................................................................................****X | 34060.600 | | 26207.100 | .............................................................**X | 26207.100 | | 23742.400 | .......................................................*****...***....*****..............................**.......*****X | 41899.000 | | 30876.800 | ........................................................................*X | 30876.800 | | 26963.900 | ...............................................................*****X | 26963.900 | | 26562.700 | ..............................................................*X | 26562.700 | | 26331.000 | .............................................................*****X | 26331.000 | | 24604.100 | .........................................................*X | 24604.100 | | 22708.800 | .....................................................******..........**.....******............................**.*****X | 41353.800 | | 33124.300 | ..............................................................................*X | 33124.300 | | 31596.900 | ..........................................................................*****X | 31596.900 | | 27602.000 | ................................................................**X | 27602.000 | | 27269.900 | ................................................................*****......*******X | 30161.700 | | 27447.900 | ................................................................**X | 27447.900 | | 26639.900 | ..............................................................*****X | 26639.900 | | 22686.500 | .....................................................**.*****.........................***..****X | 34417.500 | | 27333.400 | ................................................................**X | 27333.400 | | 25373.700 | ...........................................................*****X | 25373.700 | | 22508.000 | ....................................................**.*****X | 22859.100 | | 22658.900 | .....................................................*X | 22658.900 | | 22350.600 | ....................................................******......**.*****X | 25341.100 | | 19642.100 | .............................................*X | 19642.100 | | 18775.700 | ...........................................******..*X | 19353.900 | | 19122.300 | ............................................******..........**..*****X | 24031.200 | | 18616.400 | ...........................................*X | 18616.400 | | 17670.800 | .........................................******......*******X | 20513.300 | | 20129.700 | ...............................................*X | 20129.700 | | 20052.600 | ..............................................*****X | 20052.600 | | 16614.900 | ......................................**X | 16614.900 | | 15869.400 | ....................................*****........***..****X | 19874.800 | | 19126.700 | ............................................**X | 19126.700 | | 18760.100 | ...........................................****X | 18760.100 | | 18565.300 | ...........................................X | 18565.300 | | 17917.800 | .........................................*X | 17917.800 | | 17350.100 | ........................................******...**..******.*X | 19768.000 | | 19235.000 | ............................................*****X | 19235.000 | | 17505.600 | ........................................*X | 17505.600 | | 17480.900 | ........................................*****.........***..******.....*******X | 24134.800 | | 21645.000 | ..................................................*X | 21645.000 | | 21132.400 | .................................................******....**.******.*X | 23765.900 | | 23323.900 | ......................................................*****X | 23323.900 | | 19582.800 | .............................................*X | 19582.800 | | 19017.800 | ............................................******....**..******....*X | 23364.400 | | 22916.700 | .....................................................*****X | 22916.700 | | 19554.600 | .............................................*******X | 19668.000 | | 19373.200 | .............................................********.........**X | 23169.300 | | 22716.600 | .....................................................****X | 22716.600 | | 21742.600 | ..................................................***..******.*******X | 23010.300 | | 21982.700 | ...................................................*X | 21982.700 | | 21833.900 | ...................................................*****X | 21833.900 | | 19951.400 | ..............................................*X | 19951.400 | | 19192.900 | ............................................*****X | 19192.900 | | 18808.500 | ...........................................**..******..*X | 20213.300 | | 19635.800 | .............................................*****X | 19635.800 | | 18761.700 | ...........................................**.******.**X | 19672.300 | | 19273.500 | ............................................*****.......***..****X | 22881.600 | | 20781.300 | ................................................**X | 20781.300 | | 20028.900 | ..............................................***X | 20028.900 | +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 15:52:34,081 | test | INFO | pool-11-thread-1 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------+-----------+ | 20028.900 | ...................................................................................................*X | 20028.900 | +-----------+-------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 15:52:34,217 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 15:52:44,032 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 15:52:58,273 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 15:52:58,273 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 15:53:09,095 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 15:53:09,095 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 15:53:19,971 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 15:53:19,971 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 15:53:38,305 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 44100 | 9.77 GiB | 204.23 MiB | 356.90 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 44100 | 4.88 GiB | 436.77 MiB | 653.45 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 28820250 | 2.50 GiB | 1.66 GiB | 57.13 GiB | 31.2993433437 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 15:53:38,780 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 15:53:38,780 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 15:53:45,401 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 15:53:45,401 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 15:53:51,959 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 40% 2022-02-02 15:53:51,960 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 1% 2022-02-02 15:54:48,023 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 15:54:48,437 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 15:54:48,437 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 15:55:18,582 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 15:55:18,997 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 15:55:18,997 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 15:55:49,111 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 15:55:49,516 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 15:55:49,516 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 15:58:50,960 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3d6e845a4b7395ca696d6413fe621534', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=2f35ea6f6dae8d79ab630b6934604117', u'status': u'notRunning'} - rebalance failed 2022-02-02 15:58:50,986 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 15:58:50,986 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643846329752L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:58:49.752Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 2ad6f53fa7bcd1228dd1e60096e72e44'} 2022-02-02 15:58:50,986 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643846329751L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:58:49.751Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 15:58:50,986 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643846316102L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:58:36.102Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 15:58:50,986 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643846149749L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:55:49.749Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 2ad6f53fa7bcd1228dd1e60096e72e44"} 2022-02-02 15:58:50,986 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643846149740L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T15:55:49.740Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 15:58:50,986 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643846146239L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:55:46.239Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 15:58:50,987 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643846091883L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:54:51.883Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 15:58:50,987 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643846091483L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:54:51.483Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 15:58:50,987 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643846058104L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:54:18.104Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} 2022-02-02 15:58:50,987 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643846058093L, u'shortText': u'message', u'serverTime': u'2022-02-02T15:54:18.093Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.36\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster 2022-02-02 15:58:50,989 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3d6e845a4b7395ca696d6413fe621534', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=2f35ea6f6dae8d79ab630b6934604117', u'status': u'notRunning'} - rebalance failed ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3d6e845a4b7395ca696d6413fe621534', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=2f35ea6f6dae8d79ab630b6934604117', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- During the test, Remote Connections: 271, Disconnections: 271 SDK Connections: 30, Disconnections: 30 Ran 1 test in 5408.957s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 8 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_8 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_9 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,compaction=True,bucket_spec=magma_dgm.10_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_rebalance_out_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_9', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_rebalance_out_P0_set0', 'disk_optimized_thread_settings': 'True', 'compaction': 'True', 'bucket_spec': 'magma_dgm.10_percent_dgm.5_node_2_replica_magma_512', 'case_number': 9, 'randomize_value': 'True'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 15:58:56,789 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 15:58:57,190 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 16:00:00,867 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 63.4140000343 seconds to complete 2022-02-02 16:00:00,875 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 16:00:05,966 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x8ee9c6 type=client open_count=1 channel=[id: 0xeaa8cfef, 0.0.0.0/0.0.0.0:51450] timeout=300.0>) 2022-02-02 16:00:10,009 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x8ee9c7 type=client open_count=1 channel=[id: 0x56d29496, 0.0.0.0/0.0.0.0:49405] timeout=300.0>) 2022-02-02 16:00:12,020 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x8ee9c8 type=client open_count=1 channel=[id: 0x05cd7a3c, 0.0.0.0/0.0.0.0:56349] timeout=300.0>) 2022-02-02 16:00:14,032 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x8ee9c9 type=client open_count=1 channel=[id: 0xac4c3218, 0.0.0.0/0.0.0.0:52275] timeout=300.0>) 2022-02-02 16:00:16,042 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x8ee9ca type=client open_count=1 channel=[id: 0x0142d3a2, 0.0.0.0/0.0.0.0:48719] timeout=300.0>) 2022-02-02 16:00:21,698 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x8ee9cb type=client open_count=1 channel=[id: 0xc487531a, 0.0.0.0/0.0.0.0:38601] timeout=300.0>) 2022-02-02 16:00:23,711 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x8ee9cc type=client open_count=1 channel=[id: 0x53d2c702, 0.0.0.0/0.0.0.0:33099] timeout=300.0>) 2022-02-02 16:00:25,723 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x8ee9cd type=client open_count=1 channel=[id: 0x034cab27, 0.0.0.0/0.0.0.0:49712] timeout=300.0>) 2022-02-02 16:00:27,733 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x8ee9ce type=client open_count=1 channel=[id: 0xddd590b0, 0.0.0.0/0.0.0.0:58092] timeout=300.0>) 2022-02-02 16:00:33,204 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x8ee9cf type=client open_count=1 channel=[id: 0xc30f5e28, 0.0.0.0/0.0.0.0:42983] timeout=300.0>) 2022-02-02 16:00:35,216 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x8ee9d0 type=client open_count=1 channel=[id: 0x8c5f8cd7, 0.0.0.0/0.0.0.0:41764] timeout=300.0>) 2022-02-02 16:00:37,229 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x8ee9d1 type=client open_count=1 channel=[id: 0x40cf6d97, 0.0.0.0/0.0.0.0:48282] timeout=300.0>) 2022-02-02 16:00:39,240 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x8ee9d2 type=client open_count=1 channel=[id: 0x942a4e4b, 0.0.0.0/0.0.0.0:32784] timeout=300.0>) 2022-02-02 16:00:44,384 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x8ee9d3 type=client open_count=1 channel=[id: 0x98444f28, 0.0.0.0/0.0.0.0:46299] timeout=300.0>) 2022-02-02 16:00:46,397 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x8ee9d4 type=client open_count=1 channel=[id: 0xf0faa351, 0.0.0.0/0.0.0.0:59925] timeout=300.0>) 2022-02-02 16:00:48,408 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x8ee9d5 type=client open_count=1 channel=[id: 0x93d9380b, 0.0.0.0/0.0.0.0:35008] timeout=300.0>) 2022-02-02 16:00:50,421 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x8ee9d6 type=client open_count=1 channel=[id: 0xe1e42d84, 0.0.0.0/0.0.0.0:58073] timeout=300.0>) 2022-02-02 16:00:55,875 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x8ee9d7 type=client open_count=1 channel=[id: 0xaabe4283, 0.0.0.0/0.0.0.0:48362] timeout=300.0>) 2022-02-02 16:00:57,884 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x8ee9d8 type=client open_count=1 channel=[id: 0x78ea3db7, 0.0.0.0/0.0.0.0:46913] timeout=300.0>) 2022-02-02 16:00:59,895 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x8ee9d9 type=client open_count=1 channel=[id: 0x10f9d6ba, 0.0.0.0/0.0.0.0:44931] timeout=300.0>) 2022-02-02 16:01:01,905 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x8ee9da type=client open_count=1 channel=[id: 0x70723cf0, 0.0.0.0/0.0.0.0:53059] timeout=300.0>) 2022-02-02 16:01:03,917 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x8ee9db type=client open_count=1 channel=[id: 0xc36fd04b, 0.0.0.0/0.0.0.0:48727] timeout=300.0>) 2022-02-02 16:01:09,519 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x8ee9dc type=client open_count=1 channel=[id: 0xed3f7405, 0.0.0.0/0.0.0.0:52333] timeout=300.0>) 2022-02-02 16:01:11,529 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x8ee9dd type=client open_count=1 channel=[id: 0x771d5aed, 0.0.0.0/0.0.0.0:54197] timeout=300.0>) 2022-02-02 16:01:13,540 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x8ee9de type=client open_count=1 channel=[id: 0xa61788c6, 0.0.0.0/0.0.0.0:50313] timeout=300.0>) 2022-02-02 16:01:15,552 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x8ee9df type=client open_count=1 channel=[id: 0xd3ee56a9, 0.0.0.0/0.0.0.0:43388] timeout=300.0>) 2022-02-02 16:01:20,986 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x8ee9e0 type=client open_count=1 channel=[id: 0x8720de12, 0.0.0.0/0.0.0.0:39561] timeout=300.0>) 2022-02-02 16:01:22,996 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x8ee9e1 type=client open_count=1 channel=[id: 0x7a515f52, 0.0.0.0/0.0.0.0:60981] timeout=300.0>) 2022-02-02 16:01:25,007 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x8ee9e2 type=client open_count=1 channel=[id: 0xe3682b16, 0.0.0.0/0.0.0.0:33684] timeout=300.0>) 2022-02-02 16:01:27,019 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 16:01:41,934 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 16:01:54,431 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 16:01:54,431 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 16:01:54,467 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 16:02:07,642 | test | INFO | pool-12-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.516480762 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+-------------+--------------+ 2022-02-02 16:02:07,663 | test | INFO | pool-12-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 16:02:22,703 | test | INFO | pool-12-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 16:02:22,717 | test | INFO | pool-12-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0750000477 sec 2022-02-02 16:02:22,838 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.338430684382 | 11.45 GiB | 10.64 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.388082123185 | 11.45 GiB | 10.67 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.17986695117 | 11.45 GiB | 10.61 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.17532874139 | 11.45 GiB | 10.70 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.80632212745 | 11.45 GiB | 10.52 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 16:02:22,839 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 16:02:22,839 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 16:02:23,006 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 16:02:23,470 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 16:02:23,816 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 88.31 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 16:02:23,816 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 16:02:33,921 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 16:02:33,928 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 16:15:19,519 | test | INFO | pool-12-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.....................................................................***.......................******........*X | 65755.000 | | 62285.100 | .............................................................................................*****X | 62285.100 | | 57162.100 | .....................................................................................*X | 57162.100 | | 56859.900 | .....................................................................................*****X | 56859.900 | | 47871.600 | .......................................................................*X | 47871.600 | | 43898.300 | .................................................................*****X | 43898.300 | | 28658.000 | ..........................................*X | 28658.000 | | 26322.700 | .......................................******.......**...*****X | 32885.600 | | 21030.300 | ..............................*X | 21030.300 | | 16509.300 | ........................******.**....******...............**.*****................***........*****..................**.*****X | 58824.900 | | 37201.300 | .......................................................*X | 37201.300 | | 34780.200 | ...................................................******............................*******X | 52863.700 | | 33758.900 | ..................................................*X | 33758.900 | | 32026.400 | ...............................................******..*X | 33053.100 | | 32729.000 | ................................................******.......**.*****X | 37813.000 | | 19903.000 | .............................*X | 19903.000 | | 15033.500 | .....................*****X | 15033.500 | | 10004.500 | ..............********..**X | 11438.700 | | 11266.200 | ................*****...***.*****X | 14142.200 | | 11832.600 | ................*X | 11832.600 | | 11524.500 | ................******.*X | 12432.500 | | 12408.200 | .................******........**..******.....*******X | 22249.600 | | 17193.300 | .........................*X | 17193.300 | | 15636.900 | ......................*****X | 15636.900 | | 14579.900 | .....................********.....***.****X | 18654.100 | | 17376.700 | .........................X | 17376.700 | | 16161.000 | .......................**.******.............**....******.......*X | 32612.300 | | 30880.600 | .............................................*****X | 30880.600 | | 17013.600 | ........................*X | 17013.600 | | 15094.900 | .....................******.....**..*****X | 19354.400 | | 15767.000 | ......................*X | 15767.000 | | 15284.900 | ......................******.......**.*****X | 20392.200 | | 19284.600 | ............................*******X | 19653.500 | | 19060.600 | ...........................**X | 19060.600 | | 19029.100 | ...........................*****...********X | 20841.100 | | 16054.400 | .......................*X | 16054.400 | | 14712.800 | .....................*******X | 14963.500 | | 14870.600 | .....................*****X | 14870.600 | | 12602.300 | ..................*X | 12602.300 | | 11688.800 | ................******...............**......*****X | 25482.900 | | 24524.300 | ....................................**X | 24524.300 | | 22240.400 | ................................****X | 22240.400 | | 12125.000 | .................**X | 12125.000 | | 11279.200 | ................*****.......***.*****X | 16839.700 | | 15782.700 | .......................*X | 15782.700 | | 15629.300 | ......................******.*******X | 16170.000 | | 15270.600 | ......................*X | 15270.600 | | 14636.300 | .....................*****X | 14636.300 | | 12082.400 | .................**X | 12082.400 | | 11967.100 | .................*****.......***.****X | 17433.600 | | 17129.400 | .........................**X | 17129.400 | | 16300.800 | .......................****X | 16300.800 | | 10164.900 | ..............**X | 10164.900 | | 8812.700 | ............******...................**......*****X | 25389.200 | | 15527.400 | ......................*X | 15527.400 | | 12252.000 | .................******..********.........***...****X | 21303.000 | | 15368.400 | ......................**X | 15368.400 | | 13456.400 | ...................*****......***...****X | 19171.600 | | 17976.900 | ..........................**X | 17976.900 | | 16385.600 | .......................*****X | 16385.600 | | 11487.500 | ................*X | 11487.500 | | 10523.300 | ...............******...**..******........................**....*****X | 32624.000 | | 17868.100 | ..........................*X | 17868.100 | | 14444.400 | ....................*****X | 14444.400 | | 11667.800 | ................**X | 11667.800 | | 11165.100 | ...............****X | 11165.100 | | 11062.100 | ...............***.*****...*.........**..*****X | 20752.500 | | 15909.500 | .......................*X | 15909.500 | | 13902.500 | ....................******..........**..*****X | 22159.300 | | 9958.200 | ..............*X | 9958.200 | | 7926.400 | ...........******..........**...******.**X | 17665.000 | | 17625.200 | .........................****X | 17625.200 | | 14080.200 | ....................**X | 14080.200 | | 13823.300 | ....................*****.........********X | 20248.600 | | 9499.300 | .............*X | 9499.300 | | 7692.000 | ..........******....................**......*****X | 24345.100 | | 14616.600 | .....................*X | 14616.600 | | 11311.800 | ................******...***..*X | 14563.900 | +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 16:15:19,648 | test | INFO | pool-12-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 114.400 | .......******...............................................*X | 751.200 | | 685.600 | .................................................*****X | 685.600 | | 48.800 | ..*X | 48.800 | | 0.000 | ******................................................................***.....................*****X | 1158.900 | | 526.700 | .....................................*X | 526.700 | | 241.100 | ................*****X | 241.100 | | 0.000 | **************************...******.................................................................................**................*****X | 1359.200 | | 244.800 | .................*X | 244.800 | | 0.000 | ****************.......******..........................................................**.*****X | 902.400 | | 106.400 | ......*X | 106.400 | | 0.000 | ******************************.......................***......******.......................**......*****X | 800.000 | | 163.200 | ...........**X | 163.200 | | 0.000 | *****************************************************************.******...........................................................**.................*****X | 1047.900 | | 649.900 | ..............................................**X | 649.900 | | 431.700 | ..............................****X | 431.700 | | 33.700 | .**X | 33.700 | | 0.000 | ***************************************.....**...******...................................**........*****X | 700.000 | | 302.000 | .....................**X | 302.000 | | 200.000 | .............*******X | 200.000 | | 179.600 | ............*****X | 179.600 | | 20.400 | *X | 20.400 | | 0.000 | ***************************************************************...........***......******.........*X | 366.600 | | 346.200 | ........................*****X | 346.200 | | 174.200 | ...........*X | 174.200 | | 113.000 | .......*****X | 113.000 | | 0.000 | ****************************************************.******......****************.................**...*****X | 379.600 | | 61.200 | ...**X | 61.200 | | 0.000 | ***********************************************************************...........***...*****X | 200.000 | | 40.800 | ..*X | 40.800 | | 0.000 | *******************************************X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 16:15:19,903 | test | INFO | pool-12-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.............*****......................................................................................**X | 889.300 | | 826.400 | ...........................................................................................*****X | 826.400 | | 60.700 | .....*X | 60.700 | | 0.000 | ******........................................................................**....................*****X | 823.100 | | 251.500 | ...........................*X | 251.500 | | 76.900 | .......*****X | 76.900 | | 0.000 | **************************..******...........................................................**.........****X | 630.600 | | 106.700 | ..........**X | 106.700 | | 0.000 | ****************....*****.........................................***.****X | 411.200 | | 140.300 | ..............X | 140.300 | | 48.500 | ....*X | 48.500 | | 0.000 | ******************************....**.******..................***.....*****X | 250.000 | | 48.500 | ....*X | 48.500 | | 0.000 | ******************************************************************.******........................................**.........*****X | 450.000 | | 127.600 | .............*X | 127.600 | | 40.300 | ...*****X | 40.300 | | 0.000 | ********************************************..*****...........................***..*****X | 280.600 | | 79.100 | .......*X | 79.100 | | 50.000 | ....******.........**..*****X | 150.000 | | 29.100 | ..**X | 29.100 | | 0.000 | *************************************************.******.........*X | 95.100 | | 85.500 | ........*****X | 85.500 | | 45.200 | ....********..............***...****X | 200.000 | | 38.800 | ...**X | 38.800 | | 0.000 | *******************************************************....***.****X | 50.000 | | 9.700 | *********....*X | 50.000 | | 40.300 | ...******.....**..*****X | 100.000 | | 19.400 | .*X | 19.400 | | 0.000 | ******************************************************************.******..................***..*****X | 190.300 | | 29.100 | ..*X | 29.100 | | 0.000 | *******************************************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 16:15:21,150 | test | INFO | pool-12-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 14563.900 | ...................................................................................................X | 14563.900 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 16:15:21,298 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 37.8608247423 | 11.45 GiB | 8.96 GiB | 52.50 MiB / 3.50 GiB | 3277290 / 6576323 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 42.5490951097 | 11.45 GiB | 9.11 GiB | 24.00 MiB / 3.50 GiB | 3277051 / 6543575 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 45.7172476351 | 11.45 GiB | 9.07 GiB | 28.14 MiB / 3.50 GiB | 3283107 / 6548194 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 45.8630066037 | 11.45 GiB | 8.99 GiB | 0.0 Byte / 3.50 GiB | 3276405 / 6564011 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 38.4507401736 | 11.45 GiB | 9.13 GiB | 0.0 Byte / 3.50 GiB | 3182687 / 6360764 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2022-02-02 16:15:30,660 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 16:15:44,799 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 16:15:44,799 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 16:15:55,549 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 16:15:55,549 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 16:16:06,453 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 16:16:06,454 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 16:16:24,828 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 100000 | 9.77 GiB | 314.44 MiB | 396.25 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 508.54 MiB | 484.33 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 16287500 | 2.50 GiB | 1.68 GiB | 26.31 GiB | 23.8290345357 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 16:16:24,828 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #9 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 16:16:29,490 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_rebalance_out 2022-02-02 16:16:29,490 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_rebalance_out 2022-02-02 16:16:29,490 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:576] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 16:16:58,796 | test | INFO | pool-12-thread-9 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 29.1669998169 seconds to complete 2022-02-02 16:16:58,799 | test | INFO | pool-12-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 16:22:03,806 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait for rebalance to start 2022-02-02 16:22:13,808 | test | INFO | MainThread | [collections_rebalance:compact_all_buckets:241] Starting compaction for each bucket 2022-02-02 16:22:13,834 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 16:22:13,858 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 16:22:13,884 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 16:22:13,884 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 16:22:13,884 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 16:22:13,891 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 16:22:31,036 | test | INFO | pool-12-thread-13 | [task:call:6002] bucket1 compaction done: 100% 2022-02-02 16:22:51,086 | test | INFO | pool-12-thread-7 | [task:call:6002] bucket2 compaction done: 100% 2022-02-02 16:30:35,914 | test | INFO | pool-12-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.................**.............******..............................**..******................**.......*****X | 34040.100 | | 32731.900 | ................................................................................*X | 32731.900 | | 32000.300 | ...............................................................................*****X | 32000.300 | | 30615.900 | ...........................................................................**..*****X | 31495.300 | | 31493.900 | .............................................................................*X | 31493.900 | | 29481.800 | ........................................................................*****X | 29481.800 | | 28407.700 | ......................................................................**...*****X | 29682.600 | | 27904.700 | ....................................................................*X | 27904.700 | | 26466.500 | .................................................................******.........................**.........*****X | 39924.400 | | 31872.900 | ..............................................................................*X | 31872.900 | | 29461.500 | ........................................................................*****X | 29461.500 | | 26851.100 | ..................................................................*X | 26851.100 | | 26128.600 | ................................................................******..............***...****X | 32940.600 | | 28409.200 | ......................................................................**X | 28409.200 | | 26911.700 | ..................................................................****X | 26911.700 | | 24235.600 | ...........................................................**X | 24235.600 | | 23904.700 | ..........................................................****X | 23904.700 | | 23779.900 | ..........................................................**X | 23779.900 | | 22910.000 | ........................................................****X | 22910.000 | | 22051.600 | ......................................................**X | 22051.600 | | 21484.200 | ....................................................*****X | 21484.200 | | 20281.800 | .................................................*X | 20281.800 | | 20271.600 | .................................................******.*X | 20440.400 | | 20290.000 | .................................................*****X | 20290.000 | | 17331.100 | ..........................................********.....**X | 19276.600 | | 19064.600 | ..............................................****X | 19064.600 | | 17939.800 | ...........................................**.*****X | 18108.900 | | 18105.600 | ............................................***.*****...**X | 19822.400 | | 19387.500 | ...............................................*****X | 19387.500 | | 17983.900 | ............................................********....*******X | 19851.900 | | 15444.600 | .....................................*X | 15444.600 | | 14311.700 | ..................................******..........***..****X | 19072.200 | | 19031.700 | ..............................................X | 19031.700 | | 18942.300 | ..............................................**.*****X | 19456.000 | | 18342.000 | ............................................*X | 18342.000 | | 17414.600 | ..........................................******......**....******....*X | 23036.900 | | 22968.900 | ........................................................*****X | 22968.900 | | 19647.800 | ................................................*X | 19647.800 | | 17970.700 | ............................................******.......**.....*****X | 22913.500 | | 22648.100 | .......................................................**X | 22648.100 | | 21755.900 | .....................................................****X | 21755.900 | | 18245.800 | ............................................**X | 18245.800 | | 16803.300 | .........................................******...**...*****X | 19200.700 | | 17492.800 | ..........................................*X | 17492.800 | | 17428.600 | ..........................................******......*X | 19639.700 | | 19613.400 | ................................................******..........**.......*****X | 26709.800 | | 24626.600 | ............................................................*X | 24626.600 | | 22033.100 | ......................................................*****X | 22033.100 | | 18401.600 | .............................................**X | 18401.600 | | 18125.400 | ............................................*****......***....******..............**.*****X | 28155.300 | | 20572.000 | ..................................................*X | 20572.000 | | 18093.400 | ............................................******..............**.......*****X | 26570.200 | | 22481.000 | .......................................................*X | 22481.000 | | 19803.700 | ................................................*****X | 19803.700 | | 17895.700 | ...........................................**..******...*X | 19572.900 | | 19405.700 | ...............................................*****X | 19405.700 | | 18908.100 | ..............................................*X | 18908.100 | | 18418.100 | .............................................*****X | 18418.100 | | 16163.900 | .......................................*X | 16163.900 | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 16:30:36,878 | test | INFO | pool-12-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.....................................**.............******.*X | 153.700 | | 148.000 | ................................................******....**......******........*X | 200.700 | | 174.000 | .........................................................******................**...........*****X | 256.000 | | 132.000 | ...........................................*X | 132.000 | | 96.000 | ...............................******.................................**..............*****X | 238.400 | | 164.000 | .....................................................**X | 164.000 | | 117.900 | ......................................*****.......................***.............******...**.******.....................*X | 299.100 | | 284.500 | ..............................................................................................*****X | 284.500 | | 198.100 | .................................................................**..*****X | 204.700 | | 105.500 | ..................................**X | 105.500 | | 59.300 | ..................****X | 59.300 | | 56.800 | .................***...*****...........................***...........******.........*X | 204.000 | | 200.400 | ..................................................................*****X | 200.400 | | 80.900 | ..........................*X | 80.900 | | 59.200 | ..................******..................*X | 111.300 | | 101.200 | ................................*****X | 101.200 | | 24.700 | .......*********............................................**............*****X | 192.000 | | 138.600 | .............................................*X | 138.600 | | 111.300 | ....................................*****X | 111.300 | | 96.400 | ...............................**..******.*X | 106.100 | | 104.000 | .................................*****X | 104.000 | | 76.700 | ........................**X | 76.700 | | 73.100 | .......................*****.............***....****X | 125.500 | | 99.700 | ................................X | 99.700 | | 88.300 | ............................*X | 88.300 | | 68.000 | .....................******................**....*****X | 125.800 | | 78.200 | .........................*X | 78.200 | | 77.100 | ........................*****X | 77.100 | | 50.200 | ...............*X | 50.200 | | 36.000 | ...........******..............**...........******.............................................**....*****X | 258.900 | | 120.700 | .......................................**X | 120.700 | | 89.500 | ............................****X | 89.500 | | 88.700 | ............................**X | 88.700 | | 77.800 | .........................******.**.....*****X | 96.000 | | 93.500 | ..............................**.....******..........................*X | 186.500 | | 179.300 | ..........................................................*****X | 179.300 | | 56.400 | .................*X | 56.400 | | 32.000 | .........******.................................***.............****X | 169.100 | | 130.200 | ..........................................**X | 130.200 | | 125.100 | ........................................******..................................**.....*****X | 240.000 | | 132.400 | ...........................................*X | 132.400 | | 93.800 | ..............................*****X | 93.800 | | 66.900 | .....................**......******........................**...******........*X | 190.200 | | 188.700 | ..............................................................******.........**......*****X | 234.900 | | 181.100 | ...........................................................*X | 181.100 | | 149.100 | ................................................*****X | 149.100 | | 130.900 | ..........................................**....*****X | 142.900 | | 101.100 | ................................*X | 101.100 | | 77.100 | ........................******.**X | 80.000 | | 74.900 | ........................****X | 74.900 | | 72.000 | .......................**X | 72.000 | +---------+-----------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 16:30:37,032 | test | INFO | pool-12-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..................................**....................*****X | 109.200 | | 98.400 | ................................................*X | 98.400 | | 88.000 | ..........................................******....X | 95.900 | | 95.900 | ..............................................*...******.........**...******.......**..*****X | 143.400 | | 92.300 | .............................................*X | 92.300 | | 69.400 | .................................******.**.........*****X | 89.100 | | 78.300 | ......................................*X | 78.300 | | 71.000 | ..................................******...................................*******X | 142.400 | | 118.300 | .........................................................*X | 118.300 | | 116.700 | .........................................................*****X | 116.700 | | 82.000 | .......................................**X | 82.000 | | 73.500 | ...................................*****.............................................***...................****X | 200.600 | | 77.600 | .....................................**X | 77.600 | | 18.700 | ........*****.............***............*****X | 68.600 | | 66.200 | ................................*X | 66.200 | | 61.400 | .............................******...................**......*****X | 112.000 | | 44.900 | .....................**X | 44.900 | | 25.300 | ...........*****.........................***...*****X | 80.700 | | 30.000 | .............*X | 30.000 | | 25.300 | ...........******...............**..******.......*X | 73.400 | | 70.700 | ..................................*****X | 70.700 | | 60.600 | .............................**..*****X | 66.000 | | 25.400 | ...........*X | 25.400 | | 15.200 | ......******.............**X | 41.800 | | 39.300 | ..................*****.................***...........****X | 95.700 | | 50.000 | .......................**X | 50.000 | | 30.900 | ..............******................**..*****X | 67.400 | | 46.200 | ......................*X | 46.200 | | 41.500 | ...................*****X | 41.500 | | 25.100 | ...........*X | 25.100 | | 24.200 | ...........******......................**........*****X | 84.500 | | 76.500 | .....................................**X | 76.500 | | 69.900 | .................................****X | 69.900 | | 35.400 | ................***....*****..............................**X | 104.000 | | 82.200 | .......................................****X | 82.200 | | 65.500 | ...............................X | 65.500 | | 29.300 | .............**...******.......**...******.......................**....*****X | 109.900 | | 32.700 | ...............*X | 32.700 | | 12.500 | .....******...............**...*********.....*****.................**X | 92.700 | | 87.300 | ..........................................*****X | 87.300 | | 50.200 | ........................*X | 50.200 | | 46.000 | .....................*****X | 46.000 | | 30.200 | ..............**.******..............**..*****X | 66.000 | | 52.600 | .........................*X | 52.600 | | 50.500 | ........................******................................**.........*****X | 132.500 | | 56.200 | ...........................*X | 56.200 | | 29.500 | .............*****X | 29.500 | | 13.000 | .....**..******......................**....*****X | 69.500 | | 47.500 | ......................*X | 47.500 | | 32.500 | ...............*****X | 32.500 | | 5.000 | .**X | 5.000 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 16:30:38,335 | test | INFO | pool-12-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 15955.500 | ...................................................................................................X | 15955.500 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 16:30:39,525 | test | INFO | pool-12-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------+--------+ | 80.000 | ...................................................................................................*X | 80.000 | +--------+-------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 16:30:39,638 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 16:30:39,660 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 16:30:47,289 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 16:30:47,296 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 16:30:47,387 | test | INFO | pool-12-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 35.2587090164 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 18.5573521981 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 18.4880334304 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.312969454181 | --- OUT ---> | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 22.1828759327 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 16:30:47,428 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 16:30:57,460 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 33.33 2022-02-02 16:31:07,509 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 47.26 2022-02-02 16:31:17,553 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 64.17 2022-02-02 16:31:27,605 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 67.15 2022-02-02 16:31:37,644 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 67.91 2022-02-02 16:31:47,703 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 68.45 2022-02-02 16:31:57,747 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 68.89 2022-02-02 16:32:07,786 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 69.48 2022-02-02 16:32:17,828 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 69.76 2022-02-02 16:32:27,868 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 70.51 2022-02-02 16:32:37,905 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 71.11 2022-02-02 16:32:47,953 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 71.81 2022-02-02 16:32:58,009 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 72.47 2022-02-02 16:33:08,052 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 73.33 2022-02-02 16:33:18,092 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 74.36 2022-02-02 16:33:28,132 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 75.18 2022-02-02 16:33:38,171 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 75.88 2022-02-02 16:33:48,220 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 76.59 2022-02-02 16:33:58,262 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 77.13 2022-02-02 16:34:08,303 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 77.45 2022-02-02 16:34:18,346 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 78.21 2022-02-02 16:34:28,394 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 79.13 2022-02-02 16:34:38,434 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 79.78 2022-02-02 16:34:48,476 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 80.76 2022-02-02 16:34:58,515 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 81.73 2022-02-02 16:35:08,553 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 82.39 2022-02-02 16:35:18,607 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 82.93 2022-02-02 16:35:28,648 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 83.85 2022-02-02 16:35:38,686 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 84.82 2022-02-02 16:35:48,724 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 85.64 2022-02-02 16:35:58,760 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 86.45 2022-02-02 16:36:08,798 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 87.26 2022-02-02 16:36:18,842 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 88.08 2022-02-02 16:36:28,875 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 88.78 2022-02-02 16:36:38,911 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 89.49 2022-02-02 16:36:48,953 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 90.35 2022-02-02 16:36:58,990 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 91.06 2022-02-02 16:37:09,035 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 91.71 2022-02-02 16:37:19,073 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 92.52 2022-02-02 16:37:29,121 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 93.12 2022-02-02 16:37:39,165 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 93.71 2022-02-02 16:37:49,214 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 94.47 2022-02-02 16:37:59,260 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 95.07 2022-02-02 16:38:09,296 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 95.66 2022-02-02 16:38:19,334 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.04 2022-02-02 16:38:29,372 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.32 2022-02-02 16:38:39,405 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 96.69 2022-02-02 16:38:42,394 | test | INFO | pool-12-thread-28 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.............******..........................................................................**..*****X | 29661.700 | | 25576.500 | ............................................................................**.....******........**.******...*X | 31522.100 | | 31259.300 | .............................................................................................*****X | 31259.300 | | 24804.200 | .........................................................................*X | 24804.200 | | 22633.500 | ...................................................................******........**....*****X | 26824.400 | | 22811.600 | ...................................................................*X | 22811.600 | | 20615.100 | .............................................................******.....***...*****X | 23518.100 | | 17969.200 | .....................................................*X | 17969.200 | | 16474.800 | ................................................******..*X | 17098.800 | | 16871.300 | .................................................******............**.....******...............***......*****............**X | 33200.400 | | 32796.700 | .................................................................................................****X | 32796.700 | | 31195.200 | ............................................................................................***..*******X | 31860.500 | | 31579.600 | ..............................................................................................*****X | 31579.600 | | 31163.700 | ............................................................................................*X | 31163.700 | | 30761.900 | ...........................................................................................*****X | 30761.900 | | 29425.500 | .......................................................................................*X | 29425.500 | | 28312.600 | ....................................................................................*****X | 28312.600 | | 26907.000 | ................................................................................*X | 26907.000 | | 26737.900 | ...............................................................................*****X | 26737.900 | | 23360.500 | .....................................................................**.******..*X | 24487.200 | | 23425.200 | .....................................................................*****X | 23425.200 | | 22554.700 | ..................................................................***...*****........**X | 25994.200 | | 25483.500 | ...........................................................................*****X | 25483.500 | | 25113.200 | ..........................................................................*X | 25113.200 | | 25071.100 | ..........................................................................*****X | 25071.100 | | 22728.000 | ...................................................................**...*****X | 23696.600 | | 22655.300 | ...................................................................*X | 22655.300 | | 21479.900 | ...............................................................******............***.****X | 25781.500 | | 20356.400 | ............................................................***..******...............*X | 26122.400 | | 25264.400 | ...........................................................................*****X | 25264.400 | | 24721.400 | .........................................................................**..*****X | 25342.100 | | 18428.300 | ......................................................*X | 18428.300 | | 16325.600 | ................................................******....**..*****X | 18304.700 | | 17582.300 | ...................................................***.*****.**X | 18196.000 | | 18112.200 | .....................................................******..**.******.................**....*****X | 26151.600 | | 23355.500 | .....................................................................*X | 23355.500 | | 22958.800 | ....................................................................*****X | 22958.800 | | 20384.300 | ............................................................*X | 20384.300 | | 18670.800 | .......................................................*****X | 18670.800 | | 17168.100 | ..................................................**X | 17168.100 | | 16993.900 | ..................................................*****.***..****X | 18008.400 | | 16065.800 | ...............................................**X | 16065.800 | | 15233.700 | ............................................******.....**...******.*X | 17975.700 | | 16976.700 | ..................................................******.**..*****X | 18107.800 | | 17106.600 | ..................................................*X | 17106.600 | | 16845.300 | .................................................******......*X | 18855.000 | | 18118.400 | .....................................................*****X | 18118.400 | | 15745.100 | ..............................................**..****X | 16441.400 | +-----------+------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 16:38:42,417 | test | INFO | pool-12-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....................******...............................................................................*X | 346.500 | | 291.900 | ...................................................................................*****X | 291.900 | | 76.500 | .....................********....................**............*****X | 187.800 | | 185.100 | ....................................................*X | 185.100 | | 160.500 | .............................................*****X | 160.500 | | 114.900 | ................................***......*****............................................**X | 289.200 | | 286.500 | .................................................................................*****X | 286.500 | | 181.200 | ...................................................*X | 181.200 | | 152.700 | ...........................................*****X | 152.700 | | 121.200 | .................................**..*****X | 126.300 | | 32.400 | ........**X | 32.400 | | 0.000 | *****...............*........**...............******.......*X | 157.200 | | 126.000 | ...................................******................**.........*****X | 214.200 | | 183.500 | ...................................................*X | 183.500 | | 178.800 | ..................................................******..........***...****X | 222.000 | | 127.700 | ...................................**X | 127.700 | | 105.900 | .............................*****..............................***......****X | 229.800 | | 210.600 | ...........................................................***..*****...............**X | 269.100 | | 256.200 | ........................................................................*****X | 256.200 | | 162.200 | .............................................*X | 162.200 | | 156.800 | ............................................*****X | 156.800 | | 143.900 | ........................................*X | 143.900 | | 130.600 | ....................................******...**.....******............********.*X | 203.100 | | 189.400 | .....................................................*****X | 189.400 | | 66.500 | ..................**X | 66.500 | | 38.400 | ..........*****....***........******..............*X | 129.000 | | 98.900 | ...........................*****X | 98.900 | | 90.200 | .........................**..........******...**.......******............................................**X | 312.900 | | 278.500 | ...............................................................................****X | 278.500 | | 141.100 | .......................................**X | 141.100 | | 128.200 | ...................................*****X | 128.200 | | 60.000 | ................**...******..............*X | 120.000 | | 94.200 | ..........................*****X | 94.200 | | 89.300 | ........................**.......******..***.*****........***.....*****X | 168.100 | | 99.900 | ...........................*X | 99.900 | | 87.800 | ........................******...............*X | 139.600 | | 112.200 | ...............................*****X | 112.200 | | 72.200 | ...................**........******..*X | 105.500 | | 82.900 | ......................*****X | 82.900 | | 61.600 | ................**X | 61.600 | | 60.000 | ................*****.............***........*****X | 133.300 | | 83.800 | .......................*X | 83.800 | | 57.700 | ...............******....**....******......*X | 107.100 | | 100.100 | ...........................*****X | 100.100 | | 64.700 | .................********.....................**...****X | 146.000 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 16:38:42,434 | test | INFO | pool-12-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.******....................**.....*****X | 36.200 | | 9.000 | .....*X | 9.000 | | 0.000 | ******...........................**.............*****X | 55.600 | | 45.300 | ................................*X | 45.300 | | 44.900 | ................................******...............**.........******...................**X | 104.100 | | 101.400 | .........................................................................****X | 101.400 | | 90.000 | .................................................................X | 90.000 | | 85.500 | .............................................................*X | 85.500 | | 70.200 | ..................................................*****X | 70.200 | | 54.300 | ......................................**.......*****X | 63.100 | | 36.600 | .........................*X | 36.600 | | 22.800 | ...............******............***..............******...................*X | 83.600 | | 79.000 | .........................................................******.......**....******.**.......*****X | 105.400 | | 89.500 | ................................................................*X | 89.500 | | 73.400 | .....................................................****X | 73.400 | | 73.400 | .....................................................*.............***..........****X | 105.100 | | 76.600 | .......................................................**X | 76.600 | | 60.500 | ...........................................*****............................***.............*****...............********X | 135.900 | | 83.800 | ............................................................*X | 83.800 | | 75.000 | ......................................................*****X | 75.000 | | 57.800 | .........................................*X | 57.800 | | 39.400 | ...........................******..............**.............*****X | 75.700 | | 73.300 | ....................................................*X | 73.300 | | 64.500 | ..............................................******.........**....*****X | 81.700 | | 61.800 | ............................................**.******..............................***............****X | 120.000 | | 118.400 | ......................................................................................**X | 118.400 | | 93.500 | ...................................................................*****X | 93.500 | | 37.000 | ..........................*X | 37.000 | | 36.200 | .........................******...................................**..............******.....*X | 109.800 | | 92.600 | ...................................................................******.**..............*****X | 113.400 | | 63.400 | .............................................**X | 63.400 | | 54.600 | .......................................******.............................*X | 94.800 | | 82.600 | ...........................................................*****X | 82.600 | | 65.900 | ...............................................**.....*****X | 72.400 | | 28.600 | ....................*X | 28.600 | | 13.300 | ........******................***..............*****............**X | 69.400 | | 57.900 | .........................................******........**......*****X | 76.200 | | 42.000 | .............................*X | 42.000 | | 38.600 | ...........................*****X | 38.600 | | 37.000 | ..........................*X | 37.000 | | 30.100 | .....................******...............***......*****..............***.......*****X | 88.200 | | 60.600 | ...........................................*X | 60.600 | | 41.500 | .............................******..........**............******........**....******.......*X | 97.000 | | 79.900 | .........................................................*****X | 79.900 | | 66.900 | ................................................**......****X | 76.000 | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 16:38:43,691 | test | INFO | pool-12-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 16441.400 | ...................................................................................................X | 16441.400 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 16:38:43,698 | test | INFO | pool-12-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------+---------+ | 146.000 | ...................................................................................................X | 146.000 | +---------+------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 16:38:43,720 | test | INFO | pool-12-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 76.000 | ...................................................................................................X | 76.000 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 16:38:49,440 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 97.24 2022-02-02 16:38:59,476 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 97.94 2022-02-02 16:39:09,509 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: running, progress: 98.7 2022-02-02 16:39:24,553 | test | INFO | pool-12-thread-17 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 16:39:24,596 | test | INFO | pool-12-thread-17 | [task:check:468] Rebalance completed with progress: 100% in 517.2099998 sec 2022-02-02 16:39:24,717 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.105.206']} 2022-02-02 16:39:29,933 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 16:39:42,614 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 16:39:42,614 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 16:39:51,292 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 16:39:51,292 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 16:40:00,063 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 16:40:00,065 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 16:40:14,930 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 92100 | 7.81 GiB | 291.88 MiB | 369.94 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 3.91 GiB | 428.49 MiB | 467.05 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 15010750 | 2.00 GiB | 1.49 GiB | 24.93 GiB | 25.3540429359 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 16:40:15,408 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 16:40:15,410 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 16:40:22,319 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 16:40:22,319 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 16:40:29,013 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 22% 2022-02-02 16:40:29,013 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 5% 2022-02-02 16:41:24,013 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 16:41:24,434 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 16:41:24,434 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 16:41:54,576 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 16:41:54,993 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 16:41:54,993 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 16:42:25,105 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 16:42:25,519 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 16:42:25,519 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 16:45:26,974 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7e9f94ae08bb2bd706c182ae74d53ee2', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e3a525cb22d9248cfb3a3b0713d0fb8d', u'status': u'notRunning'} - rebalance failed 2022-02-02 16:45:27,002 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 16:45:27,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643849125752L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:45:25.752Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket1","bucket2","default"]}}}]}.\nRebalance Operation Id = 2aa7e37c521e871dfadab39787d9606f'} 2022-02-02 16:45:27,002 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643849125751L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:45:25.751Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket1",\n "bucket2",\n "default"]}}}]\n'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643848983784L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:43:03.784Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643848945747L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:42:25.747Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 2aa7e37c521e871dfadab39787d9606f"} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643848945739L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T16:42:25.739Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\']'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643848905925L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:41:45.925Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643848904242L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:41:44.242Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643848902764L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:41:42.764Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643848854119L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:40:54.119Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} 2022-02-02 16:45:27,003 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643848854117L, u'shortText': u'message', u'serverTime': u'2022-02-02T16:40:54.117Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.36\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() 2022-02-02 16:45:27,005 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7e9f94ae08bb2bd706c182ae74d53ee2', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e3a525cb22d9248cfb3a3b0713d0fb8d', u'status': u'notRunning'} - rebalance failed File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7e9f94ae08bb2bd706c182ae74d53ee2', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e3a525cb22d9248cfb3a3b0713d0fb8d', u'status': u'notRunning'} - rebalance failed ok During the test, Remote Connections: 241, Disconnections: 241 SDK Connections: 30, Disconnections: 30 ---------------------------------------------------------------------- Ran 1 test in 2795.989s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 9 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_9 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_10 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,compaction=True,bucket_spec=magma_dgm.10_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=hard_failover_and_rebalance_out_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_10', 'nodes_init': '5', 'GROUP': 'hard_failover_and_rebalance_out_P0_set0', 'disk_optimized_thread_settings': 'True', 'compaction': 'True', 'bucket_spec': 'magma_dgm.10_percent_dgm.5_node_2_replica_magma_512', 'case_number': 10, 'randomize_value': 'True'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 16:45:32,743 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 16:45:33,112 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 16:47:42,173 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.774000168 seconds to complete 2022-02-02 16:47:42,181 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 16:47:47,269 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x9c586e type=client open_count=1 channel=[id: 0xb525723b, 0.0.0.0/0.0.0.0:59650] timeout=300.0>) 2022-02-02 16:47:51,051 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x9c586f type=client open_count=1 channel=[id: 0x580f89ca, 0.0.0.0/0.0.0.0:34435] timeout=300.0>) 2022-02-02 16:47:53,061 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x9c5870 type=client open_count=1 channel=[id: 0x9a1f20a1, 0.0.0.0/0.0.0.0:59455] timeout=300.0>) 2022-02-02 16:47:55,072 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0x9c5871 type=client open_count=1 channel=[id: 0x4a19e235, 0.0.0.0/0.0.0.0:32965] timeout=300.0>) 2022-02-02 16:47:57,082 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x9c5872 type=client open_count=1 channel=[id: 0x05586ff9, 0.0.0.0/0.0.0.0:42291] timeout=300.0>) 2022-02-02 16:48:02,612 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x9c5873 type=client open_count=1 channel=[id: 0xec8d30c4, 0.0.0.0/0.0.0.0:43642] timeout=300.0>) 2022-02-02 16:48:04,624 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x9c5874 type=client open_count=1 channel=[id: 0x16c4a220, 0.0.0.0/0.0.0.0:51788] timeout=300.0>) 2022-02-02 16:48:06,634 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0x9c5875 type=client open_count=1 channel=[id: 0xf1523c03, 0.0.0.0/0.0.0.0:42999] timeout=300.0>) 2022-02-02 16:48:08,645 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x9c5876 type=client open_count=1 channel=[id: 0x74066660, 0.0.0.0/0.0.0.0:34985] timeout=300.0>) 2022-02-02 16:48:14,033 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x9c5877 type=client open_count=1 channel=[id: 0xcf35fc23, 0.0.0.0/0.0.0.0:36590] timeout=300.0>) 2022-02-02 16:48:16,045 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x9c5878 type=client open_count=1 channel=[id: 0xc19ef93b, 0.0.0.0/0.0.0.0:52018] timeout=300.0>) 2022-02-02 16:48:18,056 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0x9c5879 type=client open_count=1 channel=[id: 0x529535f7, 0.0.0.0/0.0.0.0:33848] timeout=300.0>) 2022-02-02 16:48:20,069 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x9c587a type=client open_count=1 channel=[id: 0x368d75b6, 0.0.0.0/0.0.0.0:59182] timeout=300.0>) 2022-02-02 16:48:25,605 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x9c587b type=client open_count=1 channel=[id: 0x6ba3d07c, 0.0.0.0/0.0.0.0:34062] timeout=300.0>) 2022-02-02 16:48:27,618 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x9c587c type=client open_count=1 channel=[id: 0xce302ce7, 0.0.0.0/0.0.0.0:47535] timeout=300.0>) 2022-02-02 16:48:29,628 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0x9c587d type=client open_count=1 channel=[id: 0x27082a0f, 0.0.0.0/0.0.0.0:47757] timeout=300.0>) 2022-02-02 16:48:31,642 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x9c587e type=client open_count=1 channel=[id: 0x86e8a4a6, 0.0.0.0/0.0.0.0:44671] timeout=300.0>) 2022-02-02 16:48:43,792 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x9c587f type=client open_count=1 channel=[id: 0xaec07d7a, 0.0.0.0/0.0.0.0:37345] timeout=300.0>) 2022-02-02 16:48:45,803 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x9c5880 type=client open_count=1 channel=[id: 0x07a201e1, 0.0.0.0/0.0.0.0:56837] timeout=300.0>) 2022-02-02 16:48:47,815 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x9c5881 type=client open_count=1 channel=[id: 0x3ed48b29, 0.0.0.0/0.0.0.0:52741] timeout=300.0>) 2022-02-02 16:48:49,825 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0x9c5882 type=client open_count=1 channel=[id: 0x2e7a3b76, 0.0.0.0/0.0.0.0:50509] timeout=300.0>) 2022-02-02 16:48:51,836 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x9c5883 type=client open_count=1 channel=[id: 0x1a0adf7e, 0.0.0.0/0.0.0.0:43322] timeout=300.0>) 2022-02-02 16:48:56,706 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x9c5884 type=client open_count=1 channel=[id: 0x60ed209c, 0.0.0.0/0.0.0.0:50746] timeout=300.0>) 2022-02-02 16:48:58,717 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x9c5885 type=client open_count=1 channel=[id: 0xb08dfa30, 0.0.0.0/0.0.0.0:38398] timeout=300.0>) 2022-02-02 16:49:00,727 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0x9c5886 type=client open_count=1 channel=[id: 0xd7cbc52a, 0.0.0.0/0.0.0.0:49106] timeout=300.0>) 2022-02-02 16:49:02,739 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x9c5887 type=client open_count=1 channel=[id: 0x7a37e9bf, 0.0.0.0/0.0.0.0:46417] timeout=300.0>) 2022-02-02 16:49:08,167 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x9c5888 type=client open_count=1 channel=[id: 0x5e274ede, 0.0.0.0/0.0.0.0:42853] timeout=300.0>) 2022-02-02 16:49:10,177 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x9c5889 type=client open_count=1 channel=[id: 0x30c3b255, 0.0.0.0/0.0.0.0:58013] timeout=300.0>) 2022-02-02 16:49:12,188 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0x9c588a type=client open_count=1 channel=[id: 0x323d8317, 0.0.0.0/0.0.0.0:41588] timeout=300.0>) 2022-02-02 16:49:14,198 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 16:49:29,072 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 16:49:41,598 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 16:49:41,598 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 16:49:41,634 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 16:49:54,668 | test | INFO | pool-13-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.5168609753 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2022-02-02 16:49:54,688 | test | INFO | pool-13-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 16:50:09,726 | test | INFO | pool-13-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 16:50:09,740 | test | INFO | pool-13-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0729999542 sec 2022-02-02 16:50:09,861 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.940910801656 | 11.45 GiB | 10.64 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 1.59327562414 | 11.45 GiB | 10.62 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.2166060454 | 11.45 GiB | 10.61 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.400550757291 | 11.45 GiB | 10.70 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.07850514171 | 11.45 GiB | 10.58 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 16:50:09,862 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 16:50:09,862 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 16:50:10,040 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 16:50:10,525 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 16:50:10,849 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 118.28 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ 2022-02-02 16:50:10,849 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 16:50:20,142 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 16:50:20,148 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 17:00:19,846 | test | INFO | pool-13-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *................................................................................***....................****X | 1400.000 | | 280.000 | ...................**X | 280.000 | | 0.000 | **************.....................................**.........*****X | 650.000 | | 130.000 | ........*X | 130.000 | | 0.000 | ***************************************...........**........******........*X | 390.000 | | 280.000 | ...................****X | 280.000 | | 70.000 | ....X | 70.000 | | 0.000 | ****************.........................**.......*****X | 450.000 | | 370.000 | .........................*X | 370.000 | | 350.000 | ........................*****X | 350.000 | | 70.000 | ....**X | 70.000 | | 0.000 | **************************************..................................**........*****X | 600.000 | | 120.000 | .......*X | 120.000 | | 0.000 | ***************************************.....***..*************X | 100.000 | | 60.000 | ...*X | 60.000 | | 50.000 | ..*****X | 50.000 | | 10.000 | *X | 10.000 | | 0.000 | **************************************.........***.****X | 140.000 | | 20.000 | **X | 20.000 | | 0.000 | ******************************.....**..******...*******X | 150.000 | | 30.000 | .**X | 30.000 | | 0.000 | **************************************.....**..*****X | 100.000 | | 20.000 | *X | 20.000 | | 0.000 | ******..............***...*****X | 250.000 | | 50.000 | ..*X | 50.000 | | 0.000 | ***********************************************************************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 17:00:20,815 | test | INFO | pool-13-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *........***........*****........................................................................***............*****X | 78051.800 | | 57408.500 | ........................................................................*X | 57408.500 | | 52485.300 | ..................................................................*****X | 52485.300 | | 43502.200 | ......................................................*X | 43502.200 | | 40367.600 | ..................................................*****X | 40367.600 | | 16721.600 | ....................*X | 16721.600 | | 12318.700 | ..............******.......***....*****.................*******X | 33838.800 | | 18585.100 | ......................**X | 18585.100 | | 15385.000 | ..................*****......***.....*****................................**......******...............**..*****X | 66655.500 | | 51280.300 | ................................................................*X | 51280.300 | | 46746.700 | ..........................................................*****X | 46746.700 | | 26522.200 | ................................*X | 26522.200 | | 21991.400 | ...........................*********.*****.....*******X | 26945.400 | | 26413.500 | ................................X | 26413.500 | | 24727.600 | ..............................*******......***..*****X | 30492.800 | | 24149.300 | .............................*X | 24149.300 | | 21063.400 | .........................*****X | 21063.400 | | 20245.700 | ........................**...******......**.******...............**.****X | 40024.100 | | 30017.600 | .....................................********X | 30313.500 | | 26352.400 | ................................*X | 26352.400 | | 24393.300 | ..............................*****X | 24393.300 | | 22064.300 | ...........................*X | 22064.300 | | 21112.900 | ..........................*****X | 21112.900 | | 15957.600 | ...................*X | 15957.600 | | 15627.400 | ...................*******X | 16253.300 | | 15940.100 | ...................******.**..******...............**..*****X | 31549.600 | | 25949.900 | ................................**X | 25949.900 | | 24244.700 | ..............................*******.*****X | 25580.400 | | 23571.400 | .............................**X | 23571.400 | | 22363.400 | ...........................*****.*******X | 22718.700 | | 21626.600 | ..........................**.******.......***.*****.**..******...*X | 32853.200 | | 32169.200 | ........................................*****X | 32169.200 | | 27853.200 | ..................................*X | 27853.200 | | 27606.300 | ..................................******......***...*****...**X | 36915.800 | | 35354.400 | ............................................****X | 35354.400 | | 29064.700 | ....................................**X | 29064.700 | | 28160.400 | ...................................****X | 28160.400 | | 19337.900 | .......................**X | 19337.900 | | 17852.200 | .....................*****....**X | 20471.400 | | 20311.900 | .........................*****X | 20311.900 | | 20239.500 | ........................**..*****X | 21238.600 | | 18978.200 | .......................*X | 18978.200 | | 17651.500 | .....................******...**..******.....**.*****X | 25817.600 | | 21509.000 | ..........................*X | 21509.000 | | 19384.900 | .......................******.**..******.*X | 22132.300 | | 20915.500 | .........................*****X | 20915.500 | | 15718.900 | ...................**X | 15718.900 | | 15068.700 | ..................*****....***.****X | 18887.900 | | 16124.000 | ...................**X | 16124.000 | | 15381.900 | ..................*****....***..*****.............***..****X | 31598.800 | | 25096.900 | ...............................**X | 25096.900 | | 22788.200 | ............................****X | 22788.200 | | 21644.900 | ..........................***.****X | 22046.400 | | 20631.100 | .........................X | 20631.100 | | 20021.300 | ........................*******........*******X | 26457.800 | | 24378.000 | ..............................**X | 24378.000 | | 24136.100 | .............................*****.*****X | 24867.900 | +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:00:21,937 | test | INFO | pool-13-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 1429.200 | ..............................................................................***.....................*****X | 1800.000 | | 370.800 | ...................*X | 370.800 | | 0.000 | **************..........................**.......*****X | 600.000 | | 123.600 | .....**X | 123.600 | | 0.000 | **************************************.................***..........*****.....*X | 582.400 | | 397.000 | .....................*****X | 397.000 | | 0.000 | *****************....**.......******.......................................*X | 917.600 | | 897.000 | ................................................*****X | 897.000 | | 103.000 | ....*X | 103.000 | | 0.000 | *******************************........**...******..........................**.......*****X | 800.000 | | 164.800 | ........*X | 164.800 | | 0.000 | ***************************************........**...******.................**......******....*X | 700.000 | | 679.400 | ....................................*****X | 679.400 | | 123.600 | .....*X | 123.600 | | 0.000 | *************************.******....********........................................**.........*****X | 979.400 | | 185.400 | .........*X | 185.400 | | 0.000 | **********************........**....******....**X | 300.000 | | 279.400 | ..............****X | 279.400 | | 200.000 | ..........***.*******X | 220.600 | | 179.400 | ........*****X | 179.400 | | 20.600 | *X | 20.600 | | 0.000 | ***************************************........**.....******.........*X | 400.000 | | 358.800 | ..................*****X | 358.800 | | 41.200 | .**X | 41.200 | | 20.600 | *****....**X | 100.000 | | 79.400 | ...****X | 79.400 | | 0.000 | *********************************........**...*************X | 200.000 | | 41.200 | .**X | 41.200 | | 0.000 | *******************X | 0.000 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 17:00:23,196 | test | INFO | pool-13-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 24867.900 | ...................................................................................................X | 24867.900 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:00:23,305 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 59.9639546859 | 11.45 GiB | 9.07 GiB | 52.50 MiB / 3.50 GiB | 3268341 / 6534854 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 45.014719058 | 11.45 GiB | 9.23 GiB | 24.00 MiB / 3.50 GiB | 3251860 / 6518766 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 58.413926499 | 11.45 GiB | 9.09 GiB | 28.14 MiB / 3.50 GiB | 3263919 / 6490229 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 54.8332902559 | 11.45 GiB | 9.12 GiB | 0.0 Byte / 3.50 GiB | 3242464 / 6539187 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 45.7168504745 | 11.45 GiB | 9.14 GiB | 0.0 Byte / 3.50 GiB | 3252735 / 6476272 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2022-02-02 17:00:29,410 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 17:00:43,631 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:00:43,631 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:00:54,378 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:00:54,380 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:01:05,299 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:01:05,299 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:01:23,619 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 100000 | 9.77 GiB | 324.79 MiB | 377.11 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 509.40 MiB | 407.04 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 16287500 | 2.50 GiB | 1.71 GiB | 25.96 GiB | 23.0950974674 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 17:01:23,621 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #10 test_data_load_collections_with_hard_failover_rebalance_out ========= 2022-02-02 17:01:28,236 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during hard_failover_rebalance_out 2022-02-02 17:01:28,236 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : hard_failover_rebalance_out 2022-02-02 17:01:28,236 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:623] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 17:01:37,345 | test | INFO | pool-13-thread-28 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03600001335 seconds to complete 2022-02-02 17:01:37,348 | test | INFO | pool-13-thread-28 | [common_lib:sleep:22] Sleep 8.03600001335 seconds. Reason: Wait after rebalance complete 2022-02-02 17:06:45,394 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait for rebalance to start 2022-02-02 17:06:55,394 | test | INFO | MainThread | [collections_rebalance:compact_all_buckets:241] Starting compaction for each bucket 2022-02-02 17:06:55,421 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 17:06:55,446 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 17:06:55,470 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 17:06:55,470 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 17:06:55,471 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 17:06:55,479 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 17:07:26,601 | test | INFO | pool-13-thread-13 | [task:call:6002] bucket1 compaction done: 100% 2022-02-02 17:08:16,697 | test | INFO | pool-13-thread-16 | [task:call:6002] bucket2 compaction done: 100% 2022-02-02 17:15:48,744 | test | INFO | pool-13-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..................................................................***.................................*.*X | 410.000 | | 410.000 | ..................................................................................................**X | 410.000 | | 142.500 | .................................*X | 142.500 | | 8.000 | **************..............**......******......***.....****X | 133.400 | | 59.200 | .............**X | 59.200 | | 18.200 | ...******.............**...*****X | 84.400 | | 63.500 | ..............**.*****X | 66.700 | | 58.600 | .............********..........***..*****....**X | 125.900 | | 123.700 | .............................*****X | 123.700 | | 100.500 | .......................**..******...*X | 122.100 | | 112.600 | ..........................*****X | 112.600 | | 86.400 | ....................*X | 86.400 | | 84.300 | ...................******.*X | 86.700 | | 84.500 | ...................*****X | 84.500 | | 74.200 | .................*X | 74.200 | | 66.600 | ...............*****X | 66.600 | | 50.400 | ...........**.*****X | 54.300 | | 46.400 | ..........**X | 46.400 | | 26.300 | .....*****......***.....******..*X | 80.100 | | 79.200 | ..................*****X | 79.200 | | 58.600 | .............*X | 58.600 | | 46.100 | ..........******.........**.....*****X | 104.900 | | 63.200 | ..............*X | 63.200 | | 36.900 | .......******.***..****X | 48.200 | | 35.100 | .......**X | 35.100 | | 28.500 | .....*******X | 28.500 | | 25.300 | .....*****X | 25.300 | | 17.600 | ...**..******.....*******X | 47.400 | | 33.700 | .......**X | 33.700 | | 23.200 | ....******...**...*****X | 46.000 | | 40.500 | ........**..******.....***.****X | 70.900 | | 68.400 | ...............**X | 68.400 | | 56.800 | ............*****.......***...****X | 95.200 | | 79.700 | ..................**X | 79.700 | | 77.400 | .................*****........***...****X | 122.600 | | 75.200 | .................*X | 75.200 | | 58.600 | .............*****X | 58.600 | | 43.600 | .........**X | 43.600 | | 37.400 | ........*****.........***....****X | 90.400 | | 78.800 | ..................X | 78.800 | | 70.000 | ................*X | 70.000 | | 62.800 | ..............*****X | 62.800 | | 47.200 | ..........*X | 47.200 | | 36.800 | .......******....**....*****X | 66.000 | | 25.800 | .....*X | 25.800 | | 5.400 | ******...***....*****....********X | 53.000 | | 48.400 | ..........*X | 48.400 | | 45.800 | ..........******.**..*****X | 58.000 | | 44.800 | .........**.******..**X | 55.200 | | 40.000 | ........****X | 40.000 | | 5.200 | **X | 5.200 | | 0.000 | *****.......***....*****X | 48.000 | | 21.000 | ....*X | 21.000 | | 8.000 | *********X | 8.000 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 17:15:49,303 | test | INFO | pool-13-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....***...******........**...******.............**....******......**..******..**X | 17019.100 | | 16713.600 | ...........................................****X | 16713.600 | | 15281.500 | .......................................***..******............**..*****X | 21430.900 | | 19332.700 | ..................................................**...******.......................................**.......*****X | 37639.800 | | 32225.000 | ....................................................................................**X | 32225.000 | | 30876.100 | .................................................................................*****.........***.....*****...**X | 37451.200 | | 37207.600 | .................................................................................................*****X | 37207.600 | | 34141.700 | .........................................................................................X | 34141.700 | | 33390.800 | .......................................................................................*****X | 33390.800 | | 33237.600 | .......................................................................................********.*X | 33544.400 | | 32547.400 | .....................................................................................*****X | 32547.400 | | 20419.300 | .....................................................*X | 20419.300 | | 15377.900 | .......................................******...........**..........*****X | 23052.700 | | 16310.300 | ..........................................**X | 16310.300 | | 11992.200 | ..............................*****..........***.......*****...**X | 19515.200 | | 19352.600 | ..................................................*****X | 19352.600 | | 18712.500 | ................................................*X | 18712.500 | | 18561.100 | ................................................******...*X | 19885.800 | | 19460.900 | ..................................................*****X | 19460.900 | | 17020.700 | ............................................**.******......**.*****X | 20164.100 | | 17440.700 | .............................................**X | 17440.700 | | 15862.300 | .........................................*****.***...*****........***..*****X | 21303.200 | | 17367.400 | .............................................*X | 17367.400 | | 16538.300 | ..........................................******..**.*******X | 17387.500 | | 16976.800 | ............................................*****X | 16976.800 | | 15111.600 | .......................................********......***.****X | 17984.000 | | 17471.300 | .............................................***..*****...........********.......***.....*****............***....****X | 32809.000 | | 31725.900 | ...................................................................................**.*****X | 32294.000 | | 31099.700 | .................................................................................*X | 31099.700 | | 28739.700 | ...........................................................................*****X | 28739.700 | | 27561.100 | ........................................................................**..*****X | 28575.900 | | 26447.500 | .....................................................................*X | 26447.500 | | 24824.100 | ................................................................*****X | 24824.100 | | 23622.700 | .............................................................***.****X | 23929.900 | | 22162.000 | .........................................................**X | 22162.000 | | 20871.500 | ......................................................*****........***...*****.***.*****X | 25834.600 | | 18522.300 | ................................................*X | 18522.300 | | 15363.000 | .......................................******......**..*****X | 18306.200 | | 16766.300 | ...........................................**.******....*X | 18671.500 | | 18078.600 | ...............................................*****X | 18078.600 | | 16774.000 | ...........................................**X | 16774.000 | | 16525.200 | ..........................................****X | 16525.200 | | 15675.700 | ........................................***.*****..........***.****X | 19962.300 | | 14848.400 | ......................................**X | 14848.400 | | 13118.300 | .................................*****........***......*X | 18371.800 | +-----------+------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:15:49,313 | test | INFO | pool-13-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........................................................................***..........................*****X | 648.000 | | 162.800 | ........................*X | 162.800 | | 0.000 | ********.******.........**.******..................***..........****X | 256.000 | | 168.000 | ........................X | 168.000 | | 124.000 | ..................*X | 124.000 | | 60.000 | ........******.............**...*********..****X | 176.000 | | 116.000 | ................**X | 116.000 | | 108.000 | ...............******..........**.******.............**....*****X | 288.900 | | 214.200 | ................................*X | 214.200 | | 201.300 | ..............................*****X | 201.300 | | 189.300 | ............................*X | 189.300 | | 178.700 | ..........................******.......**.....*****X | 258.700 | | 216.000 | ................................*X | 216.000 | | 165.300 | ........................*****X | 165.300 | | 85.300 | ............*X | 85.300 | | 80.000 | ...........******..**..*****X | 108.000 | | 57.300 | .......**X | 57.300 | | 29.300 | ...******............**......*****X | 145.300 | | 124.000 | ..................*X | 124.000 | | 109.400 | ...............******......***..*****..***.*****X | 178.700 | | 81.200 | ...........*X | 81.200 | | 53.300 | .......******.......**.*****X | 104.000 | | 77.300 | ..........**X | 77.300 | | 66.700 | .........*****..***..*************X | 96.000 | | 66.700 | .........********......*X | 105.200 | | 85.300 | ............******.....***......*****X | 161.300 | | 114.400 | ................*X | 114.400 | | 102.600 | ..............******......*X | 137.500 | | 132.300 | ...................*****X | 132.300 | | 103.700 | ...............*X | 103.700 | | 95.900 | .............*****X | 95.900 | | 86.800 | ............**X | 86.800 | | 83.900 | ...........*****.........***....*****............***...****X | 263.900 | | 119.100 | .................*X | 119.100 | | 104.100 | ...............******...........*X | 181.000 | | 148.000 | .....................*****X | 148.000 | | 106.700 | ...............**....*****X | 131.800 | | 125.500 | ..................**X | 125.500 | | 96.100 | .............****X | 96.100 | | 78.600 | ...........***..*****.....***..****X | 140.100 | | 113.500 | ................**X | 113.500 | | 95.900 | .............******.....**...*****X | 144.100 | | 102.800 | ..............*X | 102.800 | | 81.100 | ...........******.....**..*****X | 124.000 | | 106.500 | ...............**X | 106.500 | | 102.900 | ..............*****X | 102.900 | | 76.300 | ..........*X | 76.300 | | 69.300 | .........******........*X | 119.700 | | 118.500 | .................*****X | 118.500 | | 56.200 | .......*X | 56.200 | | 52.100 | .......********X | 52.100 | | 48.000 | ......****X | 48.000 | | 24.200 | ..**X | 24.200 | | 16.000 | .*****.......***...*X | 80.000 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 17:15:50,552 | test | INFO | pool-13-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------+-------+ | 8.000 | ...................................................................................................X | 8.000 | +-------+------------------------------------------------------------------------------------------------------+-------+ 2022-02-02 17:15:50,572 | test | INFO | pool-13-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 18371.800 | ...................................................................................................X | 18371.800 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:15:50,581 | test | INFO | pool-13-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 80.000 | ...................................................................................................X | 80.000 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 17:15:50,680 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 17:15:50,700 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 17:15:58,854 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 17:15:58,884 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 17:15:58,956 | test | INFO | pool-13-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 56.4360262853 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 27.2033574971 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 34.7948357408 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.437937937938 | --- OUT ---> | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 30.4899720741 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 17:15:58,973 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 17:16:09,005 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 36.14 2022-02-02 17:16:19,038 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 57.19 2022-02-02 17:16:29,078 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 67.26 2022-02-02 17:16:39,115 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 68.18 2022-02-02 17:16:49,157 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 68.83 2022-02-02 17:16:59,198 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 69.37 2022-02-02 17:17:09,240 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 69.96 2022-02-02 17:17:19,290 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 70.61 2022-02-02 17:17:29,332 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 71.26 2022-02-02 17:17:39,371 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 71.91 2022-02-02 17:17:49,420 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 72.67 2022-02-02 17:17:59,467 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 73.6 2022-02-02 17:18:09,510 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 74.52 2022-02-02 17:18:19,555 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 75.28 2022-02-02 17:18:29,612 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 75.82 2022-02-02 17:18:39,650 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 76.25 2022-02-02 17:18:49,707 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 76.89 2022-02-02 17:18:59,746 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 77.81 2022-02-02 17:19:09,788 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 78.35 2022-02-02 17:19:19,825 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 79.12 2022-02-02 17:19:29,868 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 80.05 2022-02-02 17:19:39,923 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 80.92 2022-02-02 17:19:49,963 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 81.84 2022-02-02 17:20:00,002 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 82.71 2022-02-02 17:20:10,039 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 83.63 2022-02-02 17:20:20,075 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 84.61 2022-02-02 17:20:30,118 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 85.53 2022-02-02 17:20:40,154 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 86.4 2022-02-02 17:20:50,190 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 87.55 2022-02-02 17:21:00,230 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 88.42 2022-02-02 17:21:10,279 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 89.39 2022-02-02 17:21:20,319 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 90.15 2022-02-02 17:21:30,355 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 91.18 2022-02-02 17:21:40,391 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 92.05 2022-02-02 17:21:50,430 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 92.65 2022-02-02 17:22:00,469 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 93.14 2022-02-02 17:22:10,502 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 93.63 2022-02-02 17:22:20,536 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 94.23 2022-02-02 17:22:30,572 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 94.51 2022-02-02 17:22:40,608 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 94.83 2022-02-02 17:22:50,651 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 95.21 2022-02-02 17:23:00,691 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 95.53 2022-02-02 17:23:10,726 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 95.8 2022-02-02 17:23:20,762 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 96.35 2022-02-02 17:23:29,398 | test | INFO | pool-13-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........................******..........................................................................*X | 196.200 | | 152.600 | ............................................................................*****X | 152.600 | | 98.600 | .................................................**.........................******.......*X | 162.000 | | 141.800 | .......................................................................*****X | 141.800 | | 69.000 | ..................................*X | 69.000 | | 50.800 | ........................******........********.............***........*****X | 106.400 | | 76.400 | .....................................*X | 76.400 | | 60.400 | .............................******................................**.............*****X | 148.200 | | 108.200 | ......................................................*X | 108.200 | | 104.400 | ....................................................******..................*X | 140.600 | | 124.000 | ..............................................................*****X | 124.000 | | 115.400 | .........................................................**.........*****X | 133.200 | | 114.600 | .........................................................*X | 114.600 | | 98.000 | ................................................*****X | 98.000 | | 84.000 | .........................................**...******..........*X | 108.900 | | 102.000 | ..................................................*****X | 102.000 | | 70.100 | ..................................**X | 70.100 | | 62.200 | ..............................****X | 62.200 | | 49.600 | ........................**X | 49.600 | | 48.800 | .......................******...........................**.....*****X | 111.600 | | 68.700 | ..................................*X | 68.700 | | 61.300 | ..............................******............................**.....*****X | 125.900 | | 52.700 | .........................**X | 52.700 | | 32.400 | ...............*****.........***......****X | 61.900 | | 57.800 | ............................**X | 57.800 | | 54.500 | ..........................*****..................*........**.****X | 107.600 | | 58.200 | ............................**X | 58.200 | | 51.600 | .........................*****................***.........****X | 101.800 | | 94.100 | ..............................................X | 94.100 | | 89.400 | ............................................*X | 89.400 | | 68.200 | .................................****X | 68.200 | | 23.200 | ..........**X | 23.200 | | 17.200 | .......*****.................***..........****X | 69.400 | | 64.800 | ................................X | 64.800 | | 56.300 | ...........................*X | 56.300 | | 40.500 | ...................******.........**...*****X | 64.000 | | 45.200 | ......................********......*X | 58.800 | | 52.900 | .........................*****X | 52.900 | | 14.300 | ......**...******..........................**X | 72.300 | | 66.000 | ................................****X | 66.000 | | 39.800 | ...................***......******.......*X | 65.200 | | 49.400 | ........................*****X | 49.400 | | 20.900 | .........**........******.................*X | 70.600 | | 52.600 | .........................*****X | 52.600 | | 32.100 | ...............***......*******X | 44.100 | | 39.700 | ...................******.....**.*****X | 51.300 | | 28.500 | .............*X | 28.500 | | 24.700 | ...........*****X | 24.700 | | 19.000 | ........**X | 19.000 | | 11.400 | ....*****.........***.....*X | 38.000 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 17:23:30,543 | test | INFO | pool-13-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | 1235.100 | ..X | 1235.100 | | 442.200 | ******.........................**...........******........................**............******..........*X | 33323.900 | | 30130.200 | ..........................................................................*****X | 30130.200 | | 20288.100 | .................................................*X | 20288.100 | | 19705.700 | ................................................******................**.......******.................*X | 35704.000 | | 34595.400 | .....................................................................................*****X | 34595.400 | | 27929.000 | ....................................................................**.******.........**..******............*X | 37489.800 | | 37254.000 | ...........................................................................................******..**......X | 40059.400 | | 40059.400 | ..................................................................................................*.***X | 40059.400 | | 33634.200 | ..................................................................................*X | 33634.200 | | 29774.400 | .........................................................................*****X | 29774.400 | | 27923.100 | ....................................................................**.******........**.....****X | 33353.900 | | 33353.900 | ..................................................................................*...X | 34464.100 | | 34464.100 | .....................................................................................X | 34464.100 | | 32160.100 | ...............................................................................*****X | 32160.100 | | 22906.200 | ........................................................*X | 22906.200 | | 19344.900 | ...............................................*****X | 19344.900 | | 18562.700 | .............................................**...........******...............................*X | 35275.000 | | 35022.200 | ......................................................................................*****X | 35022.200 | | 23927.800 | ..........................................................*X | 23927.800 | | 23250.000 | .........................................................******................**X | 29960.500 | | 29856.600 | .........................................................................*****.***....****X | 31668.800 | | 27503.700 | ...................................................................**X | 27503.700 | | 25037.800 | .............................................................****X | 25037.800 | | 21195.700 | ...................................................***.*****....*..**.****X | 24050.200 | | 22102.600 | ......................................................*******X | 22374.700 | | 20946.800 | ...................................................**X | 20946.800 | | 20375.400 | .................................................*****.......**X | 22965.600 | | 22726.200 | .......................................................****X | 22726.200 | | 16245.200 | .......................................**X | 16245.200 | | 14469.600 | ...................................*****....***.....*****........**X | 21404.800 | | 20579.100 | ..................................................******.**.*****X | 21306.900 | | 19587.300 | ...............................................**..******.............**..*****X | 26294.700 | | 23198.700 | ........................................................*X | 23198.700 | | 21937.200 | .....................................................*****X | 21937.200 | | 21890.600 | .....................................................**.*****X | 22433.100 | | 20472.200 | ..................................................**X | 20472.200 | | 19221.700 | ..............................................*****..***.*****.***....*****X | 22316.800 | | 21517.900 | ....................................................*X | 21517.900 | | 18722.400 | .............................................*****X | 18722.400 | | 15667.700 | ......................................*******X | 15782.900 | | 15545.900 | .....................................**.******...................**..*****X | 24348.500 | | 17217.800 | .........................................**X | 17217.800 | | 16376.600 | .......................................****X | 16376.600 | | 13416.700 | ................................**X | 13416.700 | | 12318.400 | .............................*****................***.....**X | 20568.800 | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:23:30,799 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 96.35 2022-02-02 17:23:31,460 | test | INFO | pool-13-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *........................******............................................................................*X | 668.500 | | 519.600 | ............................................................................*****X | 519.600 | | 195.600 | ............................**.............******...*X | 304.100 | | 242.600 | ...................................*****X | 242.600 | | 130.600 | ..................**..******................**..*****X | 262.800 | | 170.300 | ........................*X | 170.300 | | 124.200 | .................******.......***........*****.....*..*X | 267.900 | | 255.600 | .....................................*****X | 255.600 | | 166.000 | .......................*X | 166.000 | | 157.800 | ......................******......**..******...*X | 232.200 | | 223.800 | ................................*****X | 223.800 | | 204.100 | .............................*X | 204.100 | | 176.700 | .........................******..**.....******.....*******X | 257.800 | | 156.200 | ......................*X | 156.200 | | 104.700 | ..............******.***.....*****..........*******X | 213.600 | | 174.400 | .........................***..*****....**X | 217.200 | | 187.800 | ...........................********...*****X | 210.900 | | 95.000 | .............*X | 95.000 | | 66.800 | ........******......*X | 103.700 | | 103.300 | ..............******.............***...****X | 209.400 | | 152.800 | .....................**X | 152.800 | | 145.500 | ....................****X | 145.500 | | 88.700 | ............**X | 88.700 | | 84.600 | ...........*****......*...*X | 141.600 | | 129.900 | ..................****X | 129.900 | | 36.000 | ....**X | 36.000 | | 24.300 | ..*****.........***.....******...*X | 137.100 | | 106.200 | ..............*****X | 106.200 | | 72.600 | .........**..*****X | 86.700 | | 80.700 | ...........*X | 80.700 | | 75.600 | ..........******....***..****X | 114.900 | | 103.500 | ..............**X | 103.500 | | 102.000 | ..............******....*X | 132.900 | | 114.900 | ................*****X | 114.900 | | 47.100 | ......**.******....*X | 84.300 | | 67.500 | .........*****X | 67.500 | | 19.200 | .***..*****....**X | 57.600 | | 39.600 | ....*****X | 39.600 | | 34.200 | ....**...*******X | 53.700 | | 45.900 | .....******...........**...*****X | 135.000 | | 46.500 | .....**X | 46.500 | | 14.100 | .****X | 14.100 | | 11.400 | ***.******.....**.***X | 57.300 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 17:23:32,707 | test | INFO | pool-13-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 38.000 | ...................................................................................................X | 38.000 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 17:23:32,737 | test | INFO | pool-13-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 20568.800 | ...................................................................................................X | 20568.800 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:23:32,753 | test | INFO | pool-13-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 57.300 | ...................................................................................................X | 57.300 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 17:23:40,832 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 96.62 2022-02-02 17:23:50,868 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 97.0 2022-02-02 17:24:00,898 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: running, progress: 98.37 2022-02-02 17:24:15,947 | test | INFO | pool-13-thread-22 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 17:24:15,993 | test | INFO | pool-13-thread-22 | [task:check:468] Rebalance completed with progress: 100% in 497.037999868 sec 2022-02-02 17:24:16,101 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.105.206']} 2022-02-02 17:24:21,884 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 17:24:33,940 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:24:33,940 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:24:42,650 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:24:42,650 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:24:51,438 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:24:51,438 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:25:06,234 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 92100 | 7.81 GiB | 291.47 MiB | 367.21 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 3.91 GiB | 437.79 MiB | 459.76 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 15010750 | 2.00 GiB | 1.42 GiB | 27.59 GiB | 26.0733274487 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 17:25:06,733 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 17:25:06,733 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 17:25:13,464 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 17:25:13,464 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 17:25:20,167 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 26% 2022-02-02 17:25:20,168 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 2% 2022-02-02 17:26:15,736 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 17:26:16,193 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 17:26:16,193 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 17:26:46,313 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 17:26:46,736 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 17:26:46,736 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 17:27:16,888 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 17:27:17,309 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 17:27:17,309 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 17:30:18,790 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'19398f5b716e1de389851fcd8856b323', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=9cb072eab0f9127c30db3525aff7d1d2', u'status': u'notRunning'} - rebalance failed 2022-02-02 17:30:18,819 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 17:30:18,819 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643851817512L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:30:17.512Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 35f9767160094d2ab31ed85d0fc1bc1b'} 2022-02-02 17:30:18,819 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643851817511L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:30:17.511Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 17:30:18,819 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643851637509L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:27:17.509Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 35f9767160094d2ab31ed85d0fc1bc1b"} 2022-02-02 17:30:18,821 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643851637501L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T17:27:17.501Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\']'} 2022-02-02 17:30:18,821 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643851595783L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:26:35.783Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 17:30:18,821 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643851593660L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:26:33.660Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything 2022-02-02 17:30:18,821 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643851545838L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:25:45.838Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} 2022-02-02 17:30:18,821 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643851545810L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:25:45.810Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.206\' for deletion'} 2022-02-02 17:30:18,821 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643851545808L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:25:45.808Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.33\' for deletion'} 2022-02-02 17:30:18,822 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643851545801L, u'shortText': u'message', u'serverTime': u'2022-02-02T17:25:45.801Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.36\' for deletion'} self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'19398f5b716e1de389851fcd8856b323', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=9cb072eab0f9127c30db3525aff7d1d2', u'status': u'notRunning'} - rebalance failed 2022-02-02 17:30:18,822 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'19398f5b716e1de389851fcd8856b323', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=9cb072eab0f9127c30db3525aff7d1d2', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 2691.814s OK During the test, Remote Connections: 241, Disconnections: 241 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 10 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_10 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_11 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,compaction=True,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_full_recovery_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_11', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_full_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'compaction': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 11, 'randomize_value': 'True'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 17:30:24,543 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:30:24,934 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:32:34,000 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.802000046 seconds to complete 2022-02-02 17:32:34,019 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 17:32:39,099 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xa9c716 type=client open_count=1 channel=[id: 0xa7b9cfd0, 0.0.0.0/0.0.0.0:54834] timeout=300.0>) 2022-02-02 17:32:42,918 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xa9c717 type=client open_count=1 channel=[id: 0x1cc2ea92, 0.0.0.0/0.0.0.0:60586] timeout=300.0>) 2022-02-02 17:32:44,930 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xa9c718 type=client open_count=1 channel=[id: 0x8b54dfa1, 0.0.0.0/0.0.0.0:57481] timeout=300.0>) 2022-02-02 17:32:46,940 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xa9c719 type=client open_count=1 channel=[id: 0xddebd551, 0.0.0.0/0.0.0.0:50716] timeout=300.0>) 2022-02-02 17:32:48,951 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xa9c71a type=client open_count=1 channel=[id: 0xe843ad09, 0.0.0.0/0.0.0.0:52896] timeout=300.0>) 2022-02-02 17:32:54,437 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xa9c71b type=client open_count=1 channel=[id: 0x5f90ed2e, 0.0.0.0/0.0.0.0:59696] timeout=300.0>) 2022-02-02 17:32:56,450 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xa9c71c type=client open_count=1 channel=[id: 0x925ffb7a, 0.0.0.0/0.0.0.0:42146] timeout=300.0>) 2022-02-02 17:32:58,461 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xa9c71d type=client open_count=1 channel=[id: 0xaee80d2b, 0.0.0.0/0.0.0.0:43719] timeout=300.0>) 2022-02-02 17:33:00,473 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xa9c71e type=client open_count=1 channel=[id: 0x93be0516, 0.0.0.0/0.0.0.0:51048] timeout=300.0>) 2022-02-02 17:33:06,131 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xa9c71f type=client open_count=1 channel=[id: 0x6c0a6f21, 0.0.0.0/0.0.0.0:53259] timeout=300.0>) 2022-02-02 17:33:08,142 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xa9c720 type=client open_count=1 channel=[id: 0x850b83b5, 0.0.0.0/0.0.0.0:51492] timeout=300.0>) 2022-02-02 17:33:10,155 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xa9c721 type=client open_count=1 channel=[id: 0xd060b991, 0.0.0.0/0.0.0.0:38528] timeout=300.0>) 2022-02-02 17:33:12,167 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xa9c722 type=client open_count=1 channel=[id: 0x4713d8b5, 0.0.0.0/0.0.0.0:40869] timeout=300.0>) 2022-02-02 17:33:17,492 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xa9c723 type=client open_count=1 channel=[id: 0x712aa11f, 0.0.0.0/0.0.0.0:44312] timeout=300.0>) 2022-02-02 17:33:19,503 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xa9c724 type=client open_count=1 channel=[id: 0x9be362c3, 0.0.0.0/0.0.0.0:46471] timeout=300.0>) 2022-02-02 17:33:21,513 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xa9c725 type=client open_count=1 channel=[id: 0xb208ed0d, 0.0.0.0/0.0.0.0:59754] timeout=300.0>) 2022-02-02 17:33:23,523 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xa9c726 type=client open_count=1 channel=[id: 0x289bae02, 0.0.0.0/0.0.0.0:55516] timeout=300.0>) 2022-02-02 17:33:35,303 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xa9c727 type=client open_count=1 channel=[id: 0x90a5815b, 0.0.0.0/0.0.0.0:52295] timeout=300.0>) 2022-02-02 17:33:37,315 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xa9c728 type=client open_count=1 channel=[id: 0x518c4b74, 0.0.0.0/0.0.0.0:40329] timeout=300.0>) 2022-02-02 17:33:39,325 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xa9c729 type=client open_count=1 channel=[id: 0x8713054a, 0.0.0.0/0.0.0.0:42281] timeout=300.0>) 2022-02-02 17:33:41,335 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xa9c72a type=client open_count=1 channel=[id: 0x4ca68858, 0.0.0.0/0.0.0.0:46600] timeout=300.0>) 2022-02-02 17:33:46,526 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xa9c72b type=client open_count=1 channel=[id: 0x6976768e, 0.0.0.0/0.0.0.0:40526] timeout=300.0>) 2022-02-02 17:33:48,536 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xa9c72c type=client open_count=1 channel=[id: 0xdfeb8682, 0.0.0.0/0.0.0.0:43309] timeout=300.0>) 2022-02-02 17:33:50,548 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xa9c72d type=client open_count=1 channel=[id: 0x87878cc6, 0.0.0.0/0.0.0.0:58600] timeout=300.0>) 2022-02-02 17:33:52,561 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xa9c72e type=client open_count=1 channel=[id: 0x9b77d003, 0.0.0.0/0.0.0.0:56614] timeout=300.0>) 2022-02-02 17:33:57,971 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xa9c72f type=client open_count=1 channel=[id: 0x0f371e6c, 0.0.0.0/0.0.0.0:47065] timeout=300.0>) 2022-02-02 17:33:59,983 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xa9c730 type=client open_count=1 channel=[id: 0xebce437a, 0.0.0.0/0.0.0.0:34698] timeout=300.0>) 2022-02-02 17:34:01,993 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xa9c731 type=client open_count=1 channel=[id: 0x146a761f, 0.0.0.0/0.0.0.0:54173] timeout=300.0>) 2022-02-02 17:34:04,005 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 17:34:18,887 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 17:34:31,427 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 17:34:31,427 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:34:31,446 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:34:45,137 | test | INFO | pool-14-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.70532915361 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 17:34:45,157 | test | INFO | pool-14-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 17:35:00,194 | test | INFO | pool-14-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 17:35:00,209 | test | INFO | pool-14-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0720000267 sec 2022-02-02 17:35:00,328 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.966122961104 | 11.45 GiB | 10.64 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 1.75416614459 | 11.45 GiB | 10.52 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.11417125689 | 11.45 GiB | 10.62 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.40045050682 | 11.45 GiB | 10.70 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 2.25015713388 | 11.45 GiB | 10.54 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 17:35:00,328 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:35:00,328 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:35:00,493 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 17:35:00,966 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 17:35:01,296 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 111.84 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ 2022-02-02 17:35:01,296 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 17:35:10,703 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 17:35:10,711 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 17:53:07,375 | test | INFO | pool-14-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 725.400 | ...............................................................................**....................*****X | 900.000 | | 174.600 | ..................**X | 174.600 | | 0.000 | ******************************........................................**..........****X | 450.000 | | 87.300 | ........**X | 87.300 | | 0.000 | ************************************........................................***..........****X | 450.000 | | 87.300 | ........*X | 87.300 | | 0.000 | **************************************************************...................................................................***...................****X | 779.100 | | 295.500 | ...............................**X | 295.500 | | 120.900 | ............****X | 120.900 | | 91.500 | .........X | 91.500 | | 0.000 | **************************************************************************.******.........**.*****X | 100.000 | | 59.700 | .....*X | 59.700 | | 40.300 | ...*****X | 40.300 | | 0.000 | *****************************************************************....**.******...........................**...........*****X | 388.800 | | 267.900 | ............................*X | 267.900 | | 161.200 | ................*****X | 161.200 | | 0.000 | ************************************************************************......**..******..............**....*****X | 234.200 | | 169.400 | .................*X | 169.400 | | 140.300 | ..............*****X | 140.300 | | 19.400 | .**X | 19.400 | | 0.000 | **********************************************************************....**.*****X | 50.000 | | 9.700 | **X | 9.700 | | 0.000 | *****.................***.......*****.........**X | 300.000 | | 280.600 | ..............................****X | 280.600 | | 119.400 | ............**X | 119.400 | | 100.000 | ..........****X | 100.000 | | 19.400 | .**X | 19.400 | | 0.000 | **********************************************************************************************....................*......**.......*****X | 300.000 | | 58.200 | .....*X | 58.200 | | 0.000 | **************************************************************************************************************........***....****X | 109.700 | | 69.400 | ......**X | 69.400 | | 40.300 | ...****X | 40.300 | | 0.000 | ***********************************************************X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 17:53:08,142 | test | INFO | pool-14-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *................................................................................**....................*****X | 1550.000 | | 300.700 | ..................*X | 300.700 | | 0.000 | ******************************.........................................***..........****X | 800.000 | | 155.200 | .........*X | 155.200 | | 0.000 | *************************************..........***.....******X | 238.800 | | 161.200 | .........*****X | 161.200 | | 0.000 | *********************************************************.....................................**........*****X | 700.000 | | 176.100 | ..........*X | 176.100 | | 40.300 | .*****X | 40.300 | | 0.000 | ************************************************************************.....**...******..*X | 167.500 | | 132.500 | .......*****X | 132.500 | | 51.900 | ..*X | 51.900 | | 40.300 | .*****X | 40.300 | | 0.000 | *****************************************************************..**.******...............**.....*****X | 369.400 | | 167.900 | .........*X | 167.900 | | 80.600 | ....*****X | 80.600 | | 0.000 | ********************************************************************************........**.*****X | 140.300 | | 59.700 | ..*X | 59.700 | | 50.000 | ..*****X | 50.000 | | 9.700 | *X | 9.700 | | 0.000 | ***************************************************************************************.......***...****X | 159.700 | | 79.100 | ....**X | 79.100 | | 50.000 | ..*****..*******X | 90.300 | | 69.900 | ...X | 69.900 | | 9.700 | *X | 9.700 | | 0.000 | **************************************************************************************...*....**..****X | 150.000 | | 88.500 | ....X | 88.500 | | 29.100 | *X | 29.100 | | 0.000 | **************...*X | 50.000 | | 40.300 | .*****X | 40.300 | | 0.000 | ****************************************************************************************************************************************************************************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 17:53:09,312 | test | INFO | pool-14-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....**......******............................................................................**.............*****X | 77869.600 | | 67385.500 | .....................................................................................*X | 67385.500 | | 65176.600 | ..................................................................................******..**..******......**..*****X | 74010.000 | | 71186.600 | ..........................................................................................*X | 71186.600 | | 69544.900 | ........................................................................................****X | 69544.900 | | 64102.700 | .................................................................................**X | 64102.700 | | 63981.100 | .................................................................................*****.**X | 65072.300 | | 64242.600 | .................................................................................****X | 64242.600 | | 47046.400 | ...........................................................**X | 47046.400 | | 42879.900 | ......................................................*****........**...*****X | 51895.200 | | 50717.000 | ................................................................**.*****..............**X | 62637.000 | | 62605.900 | ...............................................................................****X | 62605.900 | | 50073.400 | ...............................................................*X | 50073.400 | | 47565.100 | ............................................................*****X | 47565.100 | | 42234.500 | .....................................................*X | 42234.500 | | 41526.800 | ....................................................*******X | 41986.700 | | 41343.500 | ....................................................*****X | 41343.500 | | 28922.400 | ....................................*X | 28922.400 | | 27133.200 | .................................******.............**...*****X | 39344.000 | | 37693.200 | ...............................................*X | 37693.200 | | 36616.900 | ..............................................*****X | 36616.900 | | 34185.200 | ..........................................*X | 34185.200 | | 33684.400 | ..........................................*****X | 33684.400 | | 31090.100 | ......................................*X | 31090.100 | | 30754.800 | ......................................*****X | 30754.800 | | 30233.900 | .....................................**.******..........**..*****X | 39976.900 | | 37968.400 | ...............................................*X | 37968.400 | | 36603.100 | ..............................................*****X | 36603.100 | | 25066.300 | ...............................*X | 25066.300 | | 22540.800 | ...........................*****X | 22540.800 | | 20940.600 | .........................**.******........**..******...*******X | 31736.800 | | 27289.600 | ..................................**X | 27289.600 | | 25953.600 | ................................*****.***.****X | 27342.000 | | 26556.700 | .................................********.......***..****X | 33935.500 | | 33305.200 | .........................................**X | 33305.200 | | 31391.900 | .......................................****X | 31391.900 | | 29475.200 | ....................................**X | 29475.200 | | 29379.900 | ....................................*****X | 29379.900 | | 27101.400 | .................................**.******.......**.*****X | 34206.700 | | 29951.100 | .....................................*X | 29951.100 | | 29058.400 | ....................................*****X | 29058.400 | | 23837.300 | .............................*X | 23837.300 | | 22383.400 | ...........................******.............**....*****X | 35324.900 | | 34773.800 | ...........................................*X | 34773.800 | | 32881.800 | .........................................*****X | 32881.800 | | 27565.100 | ..................................*X | 27565.100 | | 27256.400 | ..................................*****X | 27256.400 | | 25701.100 | ................................*X | 25701.100 | | 24556.300 | ..............................*****X | 24556.300 | | 20765.800 | .........................*X | 20765.800 | | 20720.900 | .........................******........**...*****X | 29236.700 | | 25737.400 | ................................**X | 25737.400 | | 23532.300 | .............................****X | 23532.300 | | 21167.400 | ..........................**X | 21167.400 | | 21111.800 | ..........................*****.********..*...**..*****..********..............***..****X | 41532.700 | | 28248.600 | ...................................**X | 28248.600 | | 25984.700 | ................................*****.***..*****..........*******X | 36066.100 | | 35873.700 | .............................................**.*****X | 36661.100 | | 31983.700 | ........................................*X | 31983.700 | | 30041.800 | .....................................*****X | 30041.800 | | 28362.400 | ...................................**X | 28362.400 | | 27847.600 | ..................................****X | 27847.600 | | 22542.600 | ...........................**X | 22542.600 | | 21767.100 | ..........................*****.***.*****....*....**.*****X | 30277.500 | | 26198.800 | ................................*X | 26198.800 | | 25168.800 | ...............................******..**.******...*X | 30224.600 | | 30089.600 | .....................................*****X | 30089.600 | | 24701.000 | ..............................*X | 24701.000 | | 23946.100 | .............................*****X | 23946.100 | | 22754.400 | ............................*X | 22754.400 | | 22238.700 | ...........................*****X | 22238.700 | | 22115.200 | ...........................**.******........**.*****X | 30249.400 | | 23929.400 | .............................*X | 23929.400 | | 22653.300 | ............................*****X | 22653.300 | | 19742.700 | ........................*X | 19742.700 | | 19463.400 | .......................******.......**.*****X | 25042.700 | | 22576.200 | ...........................*X | 22576.200 | | 21928.500 | ...........................*****X | 21928.500 | | 18855.800 | .......................*X | 18855.800 | | 18581.700 | ......................******..........**..*****X | 27790.900 | | 24011.600 | .............................**X | 24011.600 | | 22582.500 | ............................****X | 22582.500 | | 22247.900 | ...........................*******X | 22349.900 | | 21936.000 | ...........................********X | 22060.400 | | 20158.500 | ........................*X | 20158.500 | | 20012.600 | ........................******........**...******.*X | 29047.900 | | 28199.100 | ...................................*****X | 28199.100 | | 25938.500 | ................................*******X | 26099.900 | | 23175.700 | ............................*X | 23175.700 | | 21594.400 | ..........................******..**...*****X | 25331.900 | | 23884.500 | .............................*X | 23884.500 | | 22285.300 | ...........................*****X | 22285.300 | | 22008.900 | ...........................********..**..******.*X | 26004.600 | | 24942.400 | ...............................******.*******X | 26271.900 | | 20308.600 | .........................**X | 20308.600 | | 19331.100 | .......................*****......***.*****X | 24457.100 | | 20808.800 | .........................*X | 20808.800 | | 20470.300 | .........................******.......**.*****X | 26897.200 | | 21348.800 | ..........................*X | 21348.800 | | 19800.100 | ........................******...**.******.*******X | 23729.700 | | 19497.200 | ........................*X | 19497.200 | | 18399.800 | ......................******....**.******.**.*****X | 23531.600 | | 22107.900 | ...........................*X | 22107.900 | | 20780.600 | .........................******.**..*****X | 22700.100 | | 19422.300 | .......................*X | 19422.300 | | 18248.600 | ......................******..*X | 19831.700 | | 19538.700 | ........................*****X | 19538.700 | | 17755.200 | .....................***.*****......***..*******X | 24396.500 | | 23538.000 | .............................****X | 23538.000 | | 20472.200 | .........................**X | 20472.200 | | 20381.400 | .........................*****......***.****X | 26009.600 | | 21299.400 | ..........................**X | 21299.400 | | 20508.700 | .........................*****...**X | 22694.300 | | 22634.800 | ............................****X | 22634.800 | | 20597.200 | .........................***.*****.....*.*X | 26172.600 | | 26063.500 | ................................*X | 26063.500 | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:53:10,571 | test | INFO | pool-14-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 26063.500 | ...................................................................................................X | 26063.500 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 17:53:10,671 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 57.7980472765 | 11.45 GiB | 8.67 GiB | 52.50 MiB / 3.50 GiB | 6519394 / 13030276 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 47.8449929659 | 11.45 GiB | 8.79 GiB | 24.00 MiB / 3.50 GiB | 6488779 / 13012127 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 56.1057382334 | 11.45 GiB | 8.67 GiB | 28.14 MiB / 3.50 GiB | 6536065 / 13043097 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 57.1907648652 | 11.45 GiB | 8.68 GiB | 0.0 Byte / 3.50 GiB | 6532606 / 13066234 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 36.8928296068 | 11.45 GiB | 8.85 GiB | 0.0 Byte / 3.50 GiB | 6464229 / 12925273 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 17:53:19,987 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 17:53:34,180 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:53:34,181 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:53:44,957 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:53:44,957 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:53:55,849 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 17:53:55,849 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 17:54:14,194 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 238.21 MiB | 301.48 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 502.51 MiB | 448.92 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.68 GiB | 47.92 GiB | 17.9298357636 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 17:54:14,194 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #11 test_data_load_collections_with_graceful_failover_recovery ========= 2022-02-02 17:54:18,838 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_recovery 2022-02-02 17:54:18,839 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_recovery 2022-02-02 17:54:18,839 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:670] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 17:54:48,151 | test | INFO | pool-14-thread-6 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 29.1790001392 seconds to complete 2022-02-02 17:54:48,154 | test | INFO | pool-14-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 17:59:53,155 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 17:59:53,157 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 17:59:53,163 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 18:11:48,275 | test | INFO | pool-14-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***...................................................................***.................................****X | 176.000 | | 73.400 | ........................................**X | 73.400 | | 34.400 | ..................*****...............................***......*****X | 98.800 | | 82.600 | .............................................*X | 82.600 | | 74.800 | .........................................*****X | 74.800 | | 42.400 | .......................*X | 42.400 | | 34.600 | ..................******.........**.....******.......**********.....*****X | 79.200 | | 55.100 | ..............................*X | 55.100 | | 40.000 | .....................******.........................***.........****X | 98.800 | | 84.700 | ...............................................***...****X | 90.900 | | 72.700 | ........................................**X | 72.700 | | 50.900 | ...........................*****...........**...*****X | 74.200 | | 49.700 | ...........................*X | 49.700 | | 44.000 | ........................******....***...****X | 58.000 | | 47.200 | .........................**X | 47.200 | | 44.100 | ........................******.........**...******.................**......*****X | 105.900 | | 57.700 | ...............................*X | 57.700 | | 42.100 | ......................******.........**.....******..............**..*****X | 93.500 | | 61.400 | .................................*X | 61.400 | | 40.100 | .....................*****X | 40.100 | | 10.700 | .....*X | 10.700 | | 10.600 | .....******.........**...******.*X | 34.500 | | 31.400 | ................******.......**....*****X | 50.000 | | 20.300 | ..........**X | 20.300 | | 12.600 | ......*****..........**X | 31.400 | | 29.400 | ...............****X | 29.400 | | 28.900 | ...............***....******..........**......******...............*X | 90.200 | | 82.700 | .............................................*****X | 82.700 | | 66.900 | .....................................**..*****X | 72.000 | | 26.600 | ..............*X | 26.600 | | 12.100 | .....******......................................**...........*****X | 97.900 | | 28.600 | ...............*X | 28.600 | | 10.100 | ....******.............................**........*****X | 74.500 | | 56.300 | ..............................*X | 56.300 | | 52.000 | ............................*****X | 52.000 | | 19.800 | ..........*X | 19.800 | | 5.400 | ..******.....................................***.............****X | 94.000 | | 22.400 | ...........**X | 22.400 | | 0.000 | ******...............**.....*************X | 36.000 | | 33.000 | .................**....******................................**...*****X | 100.900 | | 52.800 | .............................*X | 52.800 | | 50.700 | ...........................******.........*X | 66.500 | | 54.800 | ..............................*****X | 54.800 | | 12.200 | .....**.******....*X | 20.500 | | 13.400 | ......******.**...******...**.*****X | 28.000 | | 12.200 | .....**..******......................................**.........*****X | 97.900 | | 50.000 | ...........................**X | 50.000 | | 26.900 | ..............****X | 26.900 | | 15.900 | ........***.*****X | 18.000 | | 10.100 | ....********.......**....******......**.******............**X | 63.000 | | 55.900 | ..............................****X | 55.900 | | 51.100 | ............................***.......*****X | 64.000 | | 46.800 | .........................*X | 46.800 | | 38.500 | ....................******...........**...*****X | 62.000 | | 38.300 | ....................*X | 38.300 | | 34.500 | ..................******...**X | 40.000 | | 35.000 | ..................****X | 35.000 | | 7.500 | ...**X | 7.500 | | 2.500 | ******........*X | 15.900 | | 15.500 | .......******...........**........*****X | 48.000 | | 22.900 | ............*X | 22.900 | | 10.000 | ....******.....**......********X | 29.000 | | 21.500 | ...........*****.........***....*****X | 45.500 | | 12.500 | ......*X | 12.500 | | 0.000 | X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 18:11:48,530 | test | INFO | pool-14-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***......................................................X | 86.200 | | 86.200 | .....................................................**........................*****X | 124.000 | | 37.800 | .......................*X | 37.800 | | 15.600 | ........******............................*X | 58.800 | | 48.400 | .............................*****X | 48.400 | | 37.600 | ......................**.......********..******.................***.....****X | 85.400 | | 50.100 | ..............................**X | 50.100 | | 41.800 | .........................******...................**..*****X | 74.800 | | 47.800 | .............................*X | 47.800 | | 37.400 | ......................****X | 37.400 | | 32.000 | ...................********.......*.....**.............*****X | 71.000 | | 63.000 | .......................................*X | 63.000 | | 45.400 | ...........................******.............................................**...........................*****X | 157.200 | | 97.800 | .............................................................*X | 97.800 | | 48.400 | .............................*****X | 48.400 | | 32.200 | ...................**.....******..........**.......******.......*X | 78.100 | | 73.400 | .............................................*******X | 73.400 | | 71.300 | ............................................*****X | 71.300 | | 55.500 | ..................................*******X | 56.000 | | 31.500 | ...................*X | 31.500 | | 20.600 | ............******.........................................**.....................*****X | 119.200 | | 57.100 | ...................................**X | 57.100 | | 16.200 | .........****X | 16.200 | | 10.800 | .....***....**********************............**....*****X | 42.000 | | 28.300 | .................**.........******.......................................................*X | 129.200 | | 112.000 | ......................................................................*****X | 112.000 | | 11.000 | .....*X | 11.000 | | 5.200 | ..******.................**.....******..........**......******...**X | 70.000 | | 65.900 | ........................................****X | 65.900 | | 52.600 | ................................**X | 52.600 | | 42.200 | .........................*****..***......*****X | 54.800 | | 44.400 | ...........................*X | 44.400 | | 34.400 | ....................******....*X | 39.800 | | 38.800 | .......................******..........**........*****X | 66.500 | | 50.700 | ...............................*X | 50.700 | | 50.300 | ..............................******...................*X | 79.300 | | 73.200 | .............................................*****X | 73.200 | | 25.400 | ...............*X | 25.400 | | 18.000 | ..........******..........................**..........*****X | 74.100 | | 28.400 | .................*X | 28.400 | | 7.900 | ....******.............***.................*****......................**X | 90.500 | | 72.800 | .............................................*****X | 72.800 | | 24.200 | ..............**..******................*X | 51.900 | | 46.800 | ............................******...................**.............*****X | 96.200 | | 48.200 | .............................*X | 48.200 | | 21.300 | ............*****X | 21.300 | | 14.900 | ........**....******...*X | 26.100 | | 24.800 | ..............*****X | 24.800 | | 12.900 | .......**.******............................**.........*****X | 73.300 | | 53.800 | .................................*X | 53.800 | | 33.700 | ....................******...............***.........****X | 71.900 | | 26.900 | ................**X | 26.900 | | 16.000 | .........******.....**.*****X | 26.000 | | 15.000 | ........**..******......................*X | 52.900 | | 52.500 | ................................*****X | 52.500 | | 42.200 | .........................***.....*****............***....*****X | 75.400 | | 51.700 | ...............................*X | 51.700 | | 32.500 | ...................*****X | 32.500 | | 23.900 | ..............**.*****X | 26.000 | | 18.100 | ..........*X | 18.100 | | 16.000 | .........*****X | 16.000 | | 5.000 | ..**...******......................................***.....*****X | 78.000 | | 58.400 | ....................................*X | 58.400 | | 58.000 | ...................................*****X | 58.000 | | 22.600 | .............*X | 22.600 | | 12.600 | .......******...............**......******..............*X | 69.000 | | 64.400 | .......................................*****X | 64.400 | | 34.500 | ....................**X | 34.500 | | 32.000 | ...................****X | 32.000 | | 10.000 | .....*X | 10.000 | | 0.000 | *X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 18:11:49,730 | test | INFO | pool-14-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.........***.......*****................................................***...............*****....**X | 51261.800 | | 48039.700 | ............................................................................****X | 48039.700 | | 45885.600 | .........................................................................X | 45885.600 | | 44912.700 | .......................................................................**.*****.*.*X | 46618.700 | | 46560.700 | ..........................................................................****X | 46560.700 | | 40525.200 | ................................................................**X | 40525.200 | | 39438.200 | ..............................................................******...*X | 41300.600 | | 40339.200 | ................................................................*****X | 40339.200 | | 34061.800 | ......................................................*X | 34061.800 | | 33527.000 | .....................................................******.........**....X | 41838.400 | | 41838.400 | ..................................................................*X | 41838.400 | | 41838.400 | ..................................................................*X | 41838.400 | | 41738.000 | ..................................................................**X | 41738.000 | | 40114.300 | ...............................................................*****X | 40114.300 | | 39355.900 | ..............................................................**...*****............***...*****...........*******X | 57062.600 | | 40338.400 | ................................................................*X | 40338.400 | | 36222.500 | .........................................................*****X | 36222.500 | | 35595.800 | ........................................................******X | 35610.400 | | 35610.400 | ........................................................X | 35610.400 | | 33730.100 | .....................................................**X | 33730.100 | | 32803.500 | ....................................................*****..............**.........******.......................**.*****X | 61722.000 | | 38333.800 | .............................................................*X | 38333.800 | | 32603.200 | ...................................................*******X | 32643.300 | | 32383.500 | ...................................................******..............X | 41067.600 | | 41067.600 | .................................................................*.........*****X | 46610.200 | | 39042.100 | ..............................................................X | 39042.100 | | 39042.100 | ..............................................................X | 39042.100 | | 31054.100 | .................................................*****X | 31054.100 | | 22564.200 | ...................................*X | 22564.200 | | 21829.600 | ..................................******..**.******.....**....******..........**..*****X | 36805.500 | | 30764.500 | ................................................*X | 30764.500 | | 29879.100 | ...............................................******......................**..*****X | 44733.000 | | 28233.700 | ............................................*X | 28233.700 | | 25668.400 | ........................................******...*X | 27624.900 | | 26649.500 | ..........................................******......**....*****X | 33245.200 | | 31775.500 | ..................................................*X | 31775.500 | | 31070.600 | .................................................******........**...*****X | 37738.600 | | 32058.600 | ..................................................*X | 32058.600 | | 29135.200 | ..............................................*****X | 29135.200 | | 28421.500 | .............................................********.......**....*****X | 35677.900 | | 29990.500 | ...............................................*X | 29990.500 | | 27769.600 | ...........................................******............**....*****X | 37159.600 | | 36623.700 | ..........................................................*X | 36623.700 | | 35990.500 | .........................................................*****X | 35990.500 | | 31823.700 | ..................................................*X | 31823.700 | | 30434.800 | ................................................*****X | 30434.800 | | 26211.700 | .........................................*X | 26211.700 | | 25125.700 | .......................................******.....**.******..*X | 29661.400 | | 29399.700 | ..............................................******....**.******.*******X | 33076.000 | | 29716.900 | ...............................................*X | 29716.900 | | 29297.000 | ..............................................******.*X | 29739.500 | | 29210.500 | ..............................................*****X | 29210.500 | | 29125.700 | ..............................................*X | 29125.700 | | 29110.000 | ..............................................*****X | 29110.000 | | 25480.500 | ........................................*X | 25480.500 | | 24112.700 | ......................................*****X | 24112.700 | | 23592.100 | .....................................********.**.******.....*******X | 27923.900 | | 23738.500 | .....................................*X | 23738.500 | | 22908.300 | ....................................******..*******X | 24665.100 | | 23757.700 | .....................................*X | 23757.700 | | 23163.100 | ....................................*****X | 23163.100 | | 22824.100 | ...................................**X | 22824.100 | | 22655.200 | ...................................********..*****...********X | 25563.100 | | 23773.500 | .....................................*X | 23773.500 | | 23250.000 | ....................................******.********.**.*****X | 24692.400 | | 21903.800 | ..................................**X | 21903.800 | | 20499.700 | ................................*****...***.******....*******X | 25807.600 | | 22691.300 | ...................................*X | 22691.300 | | 21351.300 | .................................*****X | 21351.300 | | 21157.500 | .................................**..*****X | 22271.400 | | 21589.200 | .................................*X | 21589.200 | | 20482.900 | ................................*****X | 20482.900 | | 18523.900 | .............................*X | 18523.900 | | 17979.300 | ............................******...**...********X | 21671.200 | | 20708.700 | ................................****X | 20708.700 | | 20568.900 | ................................****X | 20643.600 | +-----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 18:11:51,019 | test | INFO | pool-14-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 20643.600 | ...................................................................................................X | 20643.600 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 18:11:51,131 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 18:11:51,151 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 18:11:59,654 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait for rebalance to start 2022-02-02 18:12:09,655 | test | INFO | MainThread | [collections_rebalance:compact_all_buckets:241] Starting compaction for each bucket 2022-02-02 18:12:09,681 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveAdded 2022-02-02 18:12:09,706 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveAdded 2022-02-02 18:12:09,732 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveAdded 2022-02-02 18:12:09,766 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 18:12:09,773 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 18:12:09,888 | test | INFO | pool-14-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 8.60918816866 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 6.73113148311 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 4.65203761755 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.475356517388 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 9.43610445313 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 18:12:09,910 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 18:12:19,934 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 18:12:29,967 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 7.21 2022-02-02 18:12:39,997 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 17.53 2022-02-02 18:12:50,029 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 26.36 2022-02-02 18:12:51,869 | test | INFO | pool-14-thread-10 | [task:call:6002] bucket1 compaction done: 100% 2022-02-02 18:13:00,065 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 33.86 2022-02-02 18:13:10,114 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 44.08 2022-02-02 18:13:20,148 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 52.81 2022-02-02 18:13:30,191 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 59.84 2022-02-02 18:13:40,227 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 66.61 2022-02-02 18:13:47,213 | test | INFO | pool-14-thread-28 | [task:call:6002] bucket2 compaction done: 100% 2022-02-02 18:13:50,265 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 67.13 2022-02-02 18:14:00,303 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 67.6 2022-02-02 18:14:10,339 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 68.42 2022-02-02 18:14:20,377 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 69.77 2022-02-02 18:14:30,414 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 70.41 2022-02-02 18:14:40,450 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 70.97 2022-02-02 18:14:50,487 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 71.32 2022-02-02 18:15:00,525 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 71.97 2022-02-02 18:15:10,569 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 72.79 2022-02-02 18:15:20,617 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 73.49 2022-02-02 18:15:30,661 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 73.96 2022-02-02 18:15:40,700 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 74.63 2022-02-02 18:15:50,753 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 75.1 2022-02-02 18:16:00,796 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 75.57 2022-02-02 18:16:10,845 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 76.1 2022-02-02 18:16:20,891 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 76.56 2022-02-02 18:16:30,930 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 77.12 2022-02-02 18:16:40,966 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 78.07 2022-02-02 18:16:51,006 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 78.57 2022-02-02 18:17:01,042 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 79.08 2022-02-02 18:17:11,086 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 79.68 2022-02-02 18:17:21,130 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 80.28 2022-02-02 18:17:31,173 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 80.93 2022-02-02 18:17:41,210 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 81.52 2022-02-02 18:17:51,243 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 82.3 2022-02-02 18:18:01,276 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 83.08 2022-02-02 18:18:11,312 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 83.81 2022-02-02 18:18:21,358 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 84.59 2022-02-02 18:18:31,411 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 85.37 2022-02-02 18:18:41,457 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 85.99 2022-02-02 18:18:51,493 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 86.62 2022-02-02 18:19:01,528 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 87.14 2022-02-02 18:19:11,572 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 87.76 2022-02-02 18:19:21,615 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 88.44 2022-02-02 18:19:31,654 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 89.01 2022-02-02 18:19:41,701 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 89.63 2022-02-02 18:19:51,740 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 90.21 2022-02-02 18:20:01,775 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 90.73 2022-02-02 18:20:11,809 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 91.09 2022-02-02 18:20:21,846 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 91.66 2022-02-02 18:20:31,884 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 92.24 2022-02-02 18:20:41,921 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 92.7 2022-02-02 18:20:51,963 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 93.33 2022-02-02 18:21:01,997 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 93.85 2022-02-02 18:21:12,042 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 94.32 2022-02-02 18:21:22,078 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 94.79 2022-02-02 18:21:32,118 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 95.21 2022-02-02 18:21:42,153 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 95.62 2022-02-02 18:21:52,190 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 96.15 2022-02-02 18:22:02,237 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 96.46 2022-02-02 18:22:12,283 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 96.77 2022-02-02 18:22:22,326 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 97.24 2022-02-02 18:22:32,367 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 97.55 2022-02-02 18:22:42,407 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 97.87 2022-02-02 18:22:52,447 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 98.28 2022-02-02 18:23:02,484 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 98.7 2022-02-02 18:23:12,529 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 99.06 2022-02-02 18:23:22,569 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 99.53 2022-02-02 18:23:32,601 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: running, progress: 99.79 2022-02-02 18:23:47,668 | test | INFO | pool-14-thread-19 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 18:23:47,690 | test | INFO | pool-14-thread-19 | [task:check:468] Rebalance completed with progress: 100% in 697.802000046 sec 2022-02-02 18:30:35,453 | test | INFO | pool-14-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******................................................................**....................................*****X | 294.800 | | 159.300 | .....................................................**X | 159.300 | | 55.300 | .................****X | 55.300 | | 5.900 | .***********..******..........................**....*****X | 101.300 | | 67.300 | .....................*X | 67.300 | | 62.800 | ....................**X | 62.800 | | 62.800 | ....................**X | 62.800 | | 14.200 | ...*X | 14.200 | | 0.000 | ******..................**.......**X | 73.800 | | 73.800 | ........................*X | 73.800 | | 73.800 | ........................X | 73.800 | | 62.000 | ....................*X | 62.000 | | 51.800 | ................*****X | 51.800 | | 20.800 | ......*X | 20.800 | | 18.600 | .....******..*X | 24.500 | | 22.800 | ......********X | 22.800 | | 21.100 | ......****X | 21.100 | | 15.200 | ....**X | 15.200 | | 13.500 | ...*****X | 13.500 | | 7.600 | .**.******..********..........***...****X | 53.500 | | 10.700 | ..**X | 10.700 | | 1.700 | ******.......**..******..........**...******....**X | 78.300 | | 76.100 | ........................****X | 76.100 | | 25.900 | .......**X | 25.900 | | 16.900 | ....******............**..*****X | 57.400 | | 35.400 | ...........*X | 35.400 | | 34.100 | ..........******..**X | 40.000 | | 38.300 | ...........****X | 38.300 | | 7.300 | .*X | 7.300 | | 7.300 | .X | 7.300 | | 0.000 | *****.....***.******..*******X | 24.800 | | 17.400 | ....*X | 17.400 | | 15.200 | ....*****X | 15.200 | | 9.300 | ..********......**.*****X | 30.300 | | 24.500 | .......**X | 24.500 | | 21.100 | ......*****.......**...*****X | 51.600 | | 29.500 | .........**X | 29.500 | | 22.800 | ......*****..***.*****.********...**X | 42.600 | | 39.300 | ............*****X | 39.300 | | 18.600 | .....*X | 18.600 | | 15.200 | ....*****X | 15.200 | | 9.300 | ..*X | 9.300 | | 7.600 | .******....**.******....**...*****X | 43.900 | | 39.600 | ............*X | 39.600 | | 30.500 | .........*****X | 30.500 | | 15.700 | ....*X | 15.700 | | 15.200 | ....******.....**.*****X | 34.400 | | 21.100 | ......*X | 21.100 | | 17.200 | ....*******X | 17.300 | | 17.200 | ....******....**.*****X | 32.400 | | 20.600 | .....*X | 20.600 | | 17.200 | ....*****X | 17.200 | | 9.800 | ..*X | 9.800 | | 7.600 | .******...**...******........*X | 48.100 | | 42.000 | .............*****X | 42.000 | | 31.700 | .........***.****X | 32.700 | | 17.900 | .....**X | 17.900 | | 16.900 | ....******........**.*****X | 42.300 | | 21.500 | ......*X | 21.500 | | 21.100 | ......******..*X | 28.600 | | 22.900 | ......*****X | 22.900 | | 21.300 | ......**..******...*******X | 36.600 | | 29.200 | ........*X | 29.200 | | 22.600 | ......******..*******X | 29.000 | | 9.800 | ..**.******......**X | 32.000 | | 27.200 | ........****X | 27.200 | | 2.200 | **X | 2.200 | | 0.000 | ******.....**.*****X | 19.200 | | 4.400 | *X | 4.400 | | 0.000 | ******.........**....*****X | 40.300 | | 38.700 | ............*X | 38.700 | | 30.500 | .........*****X | 30.500 | | 4.000 | **.******...............***.****X | 55.000 | | 51.900 | ................**X | 51.900 | | 49.600 | ...............*****...***...****X | 65.000 | | 35.400 | ...........**X | 35.400 | | 24.800 | .......*****....***.******.*X | 41.600 | | 40.200 | ............********..******......*X | 64.000 | | 63.400 | ....................*****X | 63.400 | | 61.800 | ...................**....*****X | 73.400 | | 66.000 | .....................*X | 66.000 | | 41.200 | ............*****X | 41.200 | | 26.400 | .......**....*******X | 38.200 | | 36.000 | ...........******...***.****X | 49.400 | | 43.600 | .............**X | 43.600 | | 36.400 | ...........****X | 36.400 | | 17.400 | ....********.**X | 19.000 | | 18.600 | .....******.*X | 22.800 | | 19.200 | .....******.......**....*****X | 51.400 | | 19.200 | .....*X | 19.200 | | 7.600 | .******....**.*****X | 21.000 | | 9.400 | ..*X | 9.400 | | 7.600 | .*****X | 7.600 | | 1.800 | **X | 1.800 | | 0.000 | *****.***.********...******......*X | 34.200 | | 26.600 | ........*****X | 26.600 | | 7.600 | .**.******.........*******X | 37.800 | | 24.600 | .......**X | 24.600 | | 21.000 | ......****X | 21.000 | | 9.400 | ..**X | 9.400 | | 5.800 | *****X | 5.800 | | 0.000 | ********.....**....******..*X | 34.000 | | 25.000 | .......*****X | 25.000 | | 1.800 | ***...*****.............**X | 49.600 | | 47.800 | ...............****X | 47.800 | | 35.800 | ...........X | 35.800 | | 24.600 | .......*X | 24.600 | | 19.200 | .....******..**..*****X | 30.400 | | 13.000 | ...*X | 13.000 | | 7.600 | .*****X | 7.600 | | 1.800 | *X | 1.800 | | 0.000 | ***************************************X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 18:30:35,594 | test | INFO | pool-14-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******..................**..............*****X | 37.300 | | 32.000 | ...........................**X | 32.000 | | 15.900 | ............****X | 15.900 | | 0.000 | *****************.....................................................................**...............................*****X | 114.000 | | 46.200 | .......................................*X | 46.200 | | 17.000 | .............*****X | 17.000 | | 11.200 | ........**X | 11.200 | | 5.900 | ....*****.....*.....**......******................................**.......*****X | 68.700 | | 43.600 | .....................................*X | 43.600 | | 39.700 | .................................*****X | 39.700 | | 22.000 | ..................*X | 22.000 | | 15.200 | ............********X | 15.200 | | 13.500 | ..........********..********...******..........................**.*****X | 49.800 | | 32.100 | ...........................*X | 32.100 | | 30.400 | .........................********..*****X | 32.100 | | 26.200 | .....................**X | 26.200 | | 22.800 | ...................******.**....*******X | 28.700 | | 27.700 | .......................******...................**X | 49.800 | | 44.700 | ......................................*****X | 44.700 | | 15.200 | ............**..******......................**..*****X | 45.400 | | 14.400 | ...........*X | 14.400 | | 9.300 | .......******..........***..*****.....***.****X | 30.400 | | 18.600 | ...............**X | 18.600 | | 15.200 | ............******.....................**....*****X | 43.900 | | 27.700 | .......................*X | 27.700 | | 26.500 | ......................*****X | 26.500 | | 13.200 | ..........*X | 13.200 | | 5.900 | ....******....................***........*******..******......**X | 47.600 | | 45.400 | ......................................****X | 45.400 | | 32.100 | ...........................***.****X | 33.900 | | 29.400 | ........................**X | 29.400 | | 23.000 | ...................*****.....***...*X | 32.900 | | 32.900 | ...........................**X | 32.900 | | 32.400 | ...........................X | 32.400 | | 28.500 | ........................*X | 28.500 | | 24.600 | ....................*****X | 24.600 | | 9.800 | .......*X | 9.800 | | 7.600 | .....******.....**...******...........*******X | 28.700 | | 22.800 | ...................**...******............*X | 41.000 | | 38.500 | ................................*****X | 38.500 | | 23.800 | ...................*X | 23.800 | | 22.800 | ...................*****X | 22.800 | | 11.000 | ........*X | 11.000 | | 7.600 | .....******...........**....*****X | 24.500 | | 18.600 | ...............*X | 18.600 | | 15.200 | ............*****X | 15.200 | | 9.300 | .......*X | 9.300 | | 5.900 | ....*****X | 5.900 | | 0.000 | ********....................***............*****............**X | 50.500 | | 43.000 | ....................................*****X | 43.000 | | 32.700 | ...........................*X | 32.700 | | 32.200 | ...........................*****X | 32.200 | | 5.600 | ...*X | 5.600 | | 0.000 | ********.******.............................**.....*****X | 40.300 | | 19.500 | ................**X | 19.500 | | 15.200 | ............*****..................***.....****X | 42.000 | | 21.300 | .................**X | 21.300 | | 19.200 | ...............*****..*.....*X | 26.500 | | 20.800 | .................******............**......*****X | 42.000 | | 9.500 | .......**....******..........................................................*X | 80.800 | | 71.600 | .............................................................*****X | 71.600 | | 5.200 | ...**X | 5.200 | | 3.400 | .*****...........................................***.......****X | 59.400 | | 10.700 | ........**X | 10.700 | | 3.400 | .******......................**..*****X | 30.700 | | 17.500 | ..............*X | 17.500 | | 15.700 | ............******.....................**.....*****X | 45.600 | | 42.500 | ....................................**.******...................................***......****X | 90.100 | | 72.400 | ..............................................................**.*****X | 73.000 | | 40.800 | ..................................*X | 40.800 | | 26.600 | ......................******..........**........******..**X | 50.000 | | 44.000 | .....................................******.**....****X | 49.700 | | 26.000 | .....................**X | 26.000 | | 15.500 | ............*****...................***.............*****...............**X | 69.200 | | 65.300 | ........................................................*****X | 65.300 | | 38.800 | .................................*X | 38.800 | | 34.400 | .............................******.**....******...*X | 44.200 | | 38.400 | ................................******.......**...*****X | 49.400 | | 23.000 | ...................**...******.................**X | 46.000 | | 36.200 | ..............................********....*****..***....****X | 47.800 | | 25.600 | .....................**X | 25.600 | | 13.200 | ..........*****.*....**.....******..*******X | 26.800 | | 6.200 | ....**.******.........................*X | 36.200 | | 33.000 | ...........................*****X | 33.000 | | 32.000 | ...........................**.....*****X | 37.800 | | 21.400 | .................*X | 21.400 | | 13.400 | ..........******..................**......*****X | 40.200 | | 32.800 | ...........................***..*****..........***...*****X | 49.400 | | 24.600 | ....................*X | 24.600 | | 15.200 | ............******..........**..*****X | 28.600 | | 26.000 | .....................**...******.............*X | 44.400 | | 44.000 | .....................................*****X | 44.000 | | 19.200 | ...............***..*****..............................***.....****X | 60.800 | | 31.800 | ..........................**X | 31.800 | | 17.400 | ..............********......******.*X | 26.200 | | 23.000 | ...................*****X | 23.000 | | 9.800 | .......**.******...........*X | 22.800 | | 21.000 | .................*****X | 21.000 | | 15.200 | ............*X | 15.200 | | 13.400 | ..........******.....***....****X | 22.800 | | 5.400 | ...**X | 5.400 | | 0.000 | ******X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 18:30:36,203 | test | INFO | pool-14-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.....................**.............******.......**X | 16563.300 | | 14210.200 | ..................................****X | 14210.200 | | 10883.400 | ..........................**X | 10883.400 | | 10859.400 | ..........................****X | 10859.400 | | 10326.800 | .........................*********..........**....*****X | 16128.600 | | 15833.500 | ......................................*X | 15833.500 | | 15385.800 | .....................................*****X | 15385.800 | | 14160.800 | ..................................*X | 14160.800 | | 13986.000 | ..................................******......**...******......*******X | 20101.500 | | 18367.900 | .............................................*X | 18367.900 | | 17913.800 | ............................................*****X | 17913.800 | | 16573.600 | ........................................********.......*X | 19166.900 | | 18942.700 | ..............................................*****X | 18942.700 | | 15140.000 | .....................................**X | 15140.000 | | 14123.400 | ..................................*****.***..*******X | 15118.700 | | 14915.600 | ....................................******......*******X | 17403.000 | | 16698.100 | .........................................*X | 16698.100 | | 16452.900 | ........................................*****X | 16452.900 | | 14628.700 | ...................................**.******...........***....*****.**X | 21023.200 | | 20080.500 | .................................................*****X | 20080.500 | | 19970.800 | .................................................*******X | 20004.500 | | 18471.400 | .............................................*X | 18471.400 | | 18333.300 | .............................................*********.*****.....***.*****X | 21022.100 | | 18182.200 | ............................................*X | 18182.200 | | 17399.400 | ..........................................******.....**...******...........*X | 24855.200 | | 23870.100 | ...........................................................*****X | 23870.100 | | 23523.200 | ..........................................................**...*****X | 24625.800 | | 24100.800 | ...........................................................*X | 24100.800 | | 23420.900 | ..........................................................******......***.****X | 26435.200 | | 21028.200 | ....................................................**X | 21028.200 | | 20567.200 | ..................................................*****.......***.****X | 23539.700 | | 22949.500 | ........................................................**X | 22949.500 | | 22692.000 | ........................................................*****.........***..******.....*X | 29255.200 | | 29123.900 | ........................................................................*****X | 29123.900 | | 18249.000 | .............................................*X | 18249.000 | | 15438.400 | .....................................******...**.******.......*******X | 19600.100 | | 19371.900 | ...............................................**..*****X | 19953.100 | | 17554.700 | ...........................................*X | 17554.700 | | 17133.000 | ..........................................******........***..****X | 21170.600 | | 17771.200 | ...........................................**X | 17771.200 | | 16535.600 | ........................................*****X | 16535.600 | | 16251.100 | .......................................**.*****X | 16286.700 | | 15177.100 | .....................................**.******........*X | 18771.400 | | 18560.800 | .............................................******.**.*****X | 19131.000 | | 17877.700 | ............................................*******X | 17969.700 | | 15936.600 | .......................................********...**.******......*X | 19872.600 | | 19330.300 | ...............................................*****X | 19330.300 | | 16863.200 | .........................................*X | 16863.200 | | 16392.500 | ........................................*****X | 16392.500 | | 16355.600 | ........................................**.******........*X | 20217.200 | | 20020.800 | .................................................*****X | 20020.800 | | 18287.300 | .............................................*******X | 18378.100 | | 14509.700 | ...................................*X | 14509.700 | | 14412.600 | ...................................******...........**.****X | 19107.900 | | 18289.000 | .............................................**X | 18289.000 | | 17722.500 | ...........................................****X | 17722.500 | | 15567.500 | ......................................***..******.....*X | 18342.700 | | 17629.800 | ...........................................******.*******X | 18032.100 | | 15551.300 | ......................................*X | 15551.300 | | 15358.700 | .....................................******........**..*****X | 19333.600 | | 19099.600 | ...............................................*X | 19099.600 | | 18376.000 | .............................................*****X | 18376.000 | | 17405.000 | ..........................................**.*******X | 17592.200 | | 17546.600 | ...........................................******.**X | 18116.100 | | 18030.500 | ............................................*****....***...*****........................***....*****...***.*****.....**..******.........*X | 39661.900 | | 39490.800 | ..................................................................................................*****X | 39490.800 | | 31653.300 | ..............................................................................*X | 31653.300 | | 29738.500 | .........................................................................*****X | 29738.500 | | 27723.100 | ....................................................................**..******...................**....*****X | 37416.700 | | 31495.400 | ..............................................................................**X | 31495.400 | | 29713.800 | .........................................................................****X | 29713.800 | | 28278.100 | ......................................................................**.******......**X | 30956.000 | | 30642.500 | ............................................................................*****X | 30642.500 | | 26246.800 | .................................................................*X | 26246.800 | | 24822.500 | .............................................................*****..***.*****....***.****X | 27884.100 | | 22928.400 | ........................................................**X | 22928.400 | | 21819.700 | ......................................................****X | 21819.700 | | 21180.300 | ....................................................**X | 21180.300 | | 21033.700 | ....................................................*****X | 21056.200 | | 21052.800 | ....................................................**.*****.....**X | 23448.700 | | 23273.300 | .........................................................****X | 23273.300 | | 22347.000 | .......................................................***..*****.......**X | 26124.000 | | 25151.200 | ..............................................................****X | 25151.200 | | 19768.400 | ................................................**X | 19768.400 | | 19206.700 | ...............................................*****X | 19206.700 | | 17830.900 | ...........................................*X | 17830.900 | | 17764.700 | ...........................................******........**...*****X | 21819.600 | | 21417.400 | ....................................................*X | 21417.400 | | 20355.500 | ..................................................*****X | 20355.500 | | 19052.300 | ...............................................**...*****.......**X | 23186.300 | | 21906.900 | ......................................................*****X | 21906.900 | | 14861.000 | ....................................*X | 14861.000 | | 14016.400 | ..................................******....**..******.....**.*****X | 18972.800 | | 16309.100 | ........................................*X | 16309.100 | | 15358.200 | .....................................*****X | 15358.200 | | 14623.200 | ...................................*X | 14623.200 | | 14615.000 | ...................................******.....*******X | 16627.500 | | 15786.800 | ......................................**X | 15786.800 | | 15775.700 | ......................................****X | 15775.700 | | 14614.700 | ...................................**X | 14614.700 | | 14347.000 | ...................................******.....*X | 16402.800 | | 16370.100 | ........................................*****X | 16370.100 | | 14846.800 | ....................................**.******.............**.*****X | 20805.100 | | 20766.700 | ...................................................*X | 20766.700 | | 20766.300 | ...................................................*****X | 20766.300 | | 14106.900 | ..................................**X | 14106.900 | | 12933.500 | ...............................*****....**X | 14410.800 | | 14259.400 | ..................................*****..***..*****X | 15658.000 | | 15632.100 | ......................................*X | 15632.100 | | 15629.400 | ......................................*******X | 15670.400 | | 15154.800 | .....................................*****X | 15154.800 | | 12453.800 | ..............................*X | 12453.800 | +-----------+--------------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 18:30:37,506 | test | INFO | pool-14-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 11496.200 | ...................................................................................................X | 11496.200 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 18:30:37,627 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 18:30:47,437 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 18:31:01,750 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 18:31:01,750 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 18:31:12,447 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 18:31:12,447 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 18:31:23,334 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 18:31:23,334 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 18:31:41,588 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 46000 | 9.77 GiB | 228.55 MiB | 309.71 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 4.88 GiB | 434.69 MiB | 471.36 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 30021250 | 2.50 GiB | 1.70 GiB | 48.98 GiB | 31.0806312196 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 18:31:42,062 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 18:31:42,062 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 18:31:48,717 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 18:31:48,717 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 18:31:55,404 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 34% 2022-02-02 18:31:55,405 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 0% 2022-02-02 18:32:50,747 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 18:32:51,167 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 18:32:51,168 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 18:33:21,305 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 18:33:21,734 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 18:33:21,736 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 18:33:51,878 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 18:33:52,285 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 18:33:52,285 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 18:36:53,709 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'24e7d00a39ff4b7387c1b0c1b7cdb599', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a23880bcec7f2cd176095b2834de9362', u'status': u'notRunning'} - rebalance failed 2022-02-02 18:36:53,734 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 18:36:53,734 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643855812497L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:36:52.497Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = e69a40223c86c6279d335ce4a6bde89e'} 2022-02-02 18:36:53,734 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643855812496L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:36:52.496Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 18:36:53,734 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643855632493L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:33:52.493Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e69a40223c86c6279d335ce4a6bde89e"} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643855632485L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T18:33:52.485Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643855583455L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:33:03.455Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643855581986L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:33:01.986Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.177', u'tstamp': 1643855580489L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:33:00.489Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643855572324L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:32:52.324Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643855571648L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:32:51.648Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 18:36:53,736 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643855540856L, u'shortText': u'message', u'serverTime': u'2022-02-02T18:32:20.856Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes 2022-02-02 18:36:53,737 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'24e7d00a39ff4b7387c1b0c1b7cdb599', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a23880bcec7f2cd176095b2834de9362', u'status': u'notRunning'} - rebalance failed return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'24e7d00a39ff4b7387c1b0c1b7cdb599', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=a23880bcec7f2cd176095b2834de9362', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 3994.854s OK During the test, Remote Connections: 259, Disconnections: 259 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 11 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_11 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_12 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,compaction=True,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=hard_failover_and_full_recovery_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_12', 'nodes_init': '5', 'GROUP': 'hard_failover_and_full_recovery_P0_set0', 'disk_optimized_thread_settings': 'True', 'compaction': 'True', 'bucket_spec': 'magma_dgm.5_percent_dgm.5_node_2_replica_magma_512', 'case_number': 12, 'randomize_value': 'True'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 18:36:59,434 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 18:36:59,812 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 18:39:08,868 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.75999999 seconds to complete 2022-02-02 18:39:08,875 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 18:39:13,971 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xbb3e29 type=client open_count=1 channel=[id: 0x7331766c, 0.0.0.0/0.0.0.0:33441] timeout=300.0>) 2022-02-02 18:39:17,801 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xbb3e2a type=client open_count=1 channel=[id: 0x9dadb72a, 0.0.0.0/0.0.0.0:54414] timeout=300.0>) 2022-02-02 18:39:19,811 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xbb3e2b type=client open_count=1 channel=[id: 0xf85819cf, 0.0.0.0/0.0.0.0:54500] timeout=300.0>) 2022-02-02 18:39:21,821 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xbb3e2c type=client open_count=1 channel=[id: 0x5159086e, 0.0.0.0/0.0.0.0:33241] timeout=300.0>) 2022-02-02 18:39:23,832 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xbb3e2d type=client open_count=1 channel=[id: 0xb180483d, 0.0.0.0/0.0.0.0:38031] timeout=300.0>) 2022-02-02 18:39:25,842 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xbb3e2e type=client open_count=1 channel=[id: 0x22f03b68, 0.0.0.0/0.0.0.0:41519] timeout=300.0>) 2022-02-02 18:39:31,535 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xbb3e2f type=client open_count=1 channel=[id: 0xc79212ca, 0.0.0.0/0.0.0.0:39640] timeout=300.0>) 2022-02-02 18:39:33,546 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xbb3e30 type=client open_count=1 channel=[id: 0xebe0d4f1, 0.0.0.0/0.0.0.0:55847] timeout=300.0>) 2022-02-02 18:39:35,555 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xbb3e31 type=client open_count=1 channel=[id: 0x68810e6d, 0.0.0.0/0.0.0.0:44845] timeout=300.0>) 2022-02-02 18:39:37,566 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xbb3e32 type=client open_count=1 channel=[id: 0x6f5ce52d, 0.0.0.0/0.0.0.0:60689] timeout=300.0>) 2022-02-02 18:39:43,033 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xbb3e33 type=client open_count=1 channel=[id: 0x71363012, 0.0.0.0/0.0.0.0:58599] timeout=300.0>) 2022-02-02 18:39:45,046 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xbb3e34 type=client open_count=1 channel=[id: 0x8979e505, 0.0.0.0/0.0.0.0:55484] timeout=300.0>) 2022-02-02 18:39:47,056 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xbb3e35 type=client open_count=1 channel=[id: 0x08451e57, 0.0.0.0/0.0.0.0:52830] timeout=300.0>) 2022-02-02 18:39:49,069 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xbb3e36 type=client open_count=1 channel=[id: 0x9fd62203, 0.0.0.0/0.0.0.0:36309] timeout=300.0>) 2022-02-02 18:39:54,186 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xbb3e37 type=client open_count=1 channel=[id: 0xe4043096, 0.0.0.0/0.0.0.0:57516] timeout=300.0>) 2022-02-02 18:39:56,197 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xbb3e38 type=client open_count=1 channel=[id: 0xd68dda84, 0.0.0.0/0.0.0.0:53777] timeout=300.0>) 2022-02-02 18:39:58,207 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xbb3e39 type=client open_count=1 channel=[id: 0xb032b0bc, 0.0.0.0/0.0.0.0:58207] timeout=300.0>) 2022-02-02 18:40:00,217 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xbb3e3a type=client open_count=1 channel=[id: 0x0948ef28, 0.0.0.0/0.0.0.0:47077] timeout=300.0>) 2022-02-02 18:40:05,727 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xbb3e3b type=client open_count=1 channel=[id: 0x6aff4c45, 0.0.0.0/0.0.0.0:39211] timeout=300.0>) 2022-02-02 18:40:07,737 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xbb3e3c type=client open_count=1 channel=[id: 0x305a5fb5, 0.0.0.0/0.0.0.0:50612] timeout=300.0>) 2022-02-02 18:40:09,769 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xbb3e3d type=client open_count=1 channel=[id: 0xe8e5570f, 0.0.0.0/0.0.0.0:49163] timeout=300.0>) 2022-02-02 18:40:11,782 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xbb3e3e type=client open_count=1 channel=[id: 0x2d244d88, 0.0.0.0/0.0.0.0:51541] timeout=300.0>) 2022-02-02 18:40:17,308 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xbb3e3f type=client open_count=1 channel=[id: 0x53709014, 0.0.0.0/0.0.0.0:34487] timeout=300.0>) 2022-02-02 18:40:19,319 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xbb3e40 type=client open_count=1 channel=[id: 0x845d5c53, 0.0.0.0/0.0.0.0:60541] timeout=300.0>) 2022-02-02 18:40:21,328 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xbb3e41 type=client open_count=1 channel=[id: 0xa3dde89e, 0.0.0.0/0.0.0.0:42772] timeout=300.0>) 2022-02-02 18:40:23,342 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xbb3e42 type=client open_count=1 channel=[id: 0x35ef37f3, 0.0.0.0/0.0.0.0:41494] timeout=300.0>) 2022-02-02 18:40:28,780 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xbb3e43 type=client open_count=1 channel=[id: 0x81e2dd72, 0.0.0.0/0.0.0.0:46406] timeout=300.0>) 2022-02-02 18:40:30,790 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xbb3e44 type=client open_count=1 channel=[id: 0x6ce6f75e, 0.0.0.0/0.0.0.0:47241] timeout=300.0>) 2022-02-02 18:40:32,801 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xbb3e45 type=client open_count=1 channel=[id: 0xe9134825, 0.0.0.0/0.0.0.0:33695] timeout=300.0>) 2022-02-02 18:40:34,811 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 18:40:49,720 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 18:41:02,178 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 18:41:02,180 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 18:41:02,197 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 18:41:15,565 | test | INFO | pool-15-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 1.55407945858 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 18:41:15,588 | test | INFO | pool-15-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 18:41:30,625 | test | INFO | pool-15-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 18:41:30,638 | test | INFO | pool-15-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0729999542 sec 2022-02-02 18:41:30,759 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 1.43072289157 | 11.45 GiB | 10.63 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 2.06430626798 | 11.45 GiB | 10.54 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 1.11514847763 | 11.45 GiB | 10.59 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.463368816531 | 11.45 GiB | 10.69 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 1.01541933058 | 11.45 GiB | 10.49 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 18:41:30,759 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 18:41:30,759 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 18:41:30,946 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 18:41:31,427 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 18:41:31,769 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 87.58 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 18:41:31,770 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 18:41:42,332 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 18:41:42,351 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 19:03:39,240 | test | INFO | pool-15-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............******..............................................................*X | 450.000 | | 360.000 | .............................................................*****X | 360.000 | | 0.000 | ********************************......**..........******..............................................................*******X | 455.900 | | 55.900 | ........*X | 55.900 | | 0.000 | *************************************..................................***.........****X | 250.000 | | 50.000 | .......**X | 50.000 | | 0.000 | *****************************************************....................**.................******...............................................................*X | 580.000 | | 530.000 | ..........................................................................................*****X | 530.000 | | 50.000 | .......**X | 50.000 | | 0.000 | ****************************************************************...******..............................................................**............*****X | 450.000 | | 170.000 | ............................*X | 170.000 | | 80.000 | ............*****X | 80.000 | | 0.000 | ***********************************************************************************.******.......*X | 50.000 | | 40.000 | .....*****X | 40.000 | | 0.000 | **...******................................................**.....*****X | 330.000 | | 50.000 | .......*X | 50.000 | | 0.000 | ***********************************************************************************************************.*****...................................***.......****X | 250.000 | | 90.000 | ..............**X | 90.000 | | 60.000 | .........******.......................................**...........******....*X | 371.700 | | 331.700 | ........................................................*****X | 331.700 | | 40.000 | .....*X | 40.000 | | 0.000 | ********************************************************************************************************************************......**..*****X | 50.000 | | 10.000 | *X | 10.000 | | 0.000 | ******......***..****X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | *******************************************************************************************************.............***....****X | 100.000 | | 20.000 | ..**X | 20.000 | | 10.000 | ******..............**.****************.*****X | 100.000 | | 20.000 | ..*X | 20.000 | | 0.000 | ******......**..*****X | 50.000 | | 10.000 | **X | 10.000 | | 0.000 | ***************************************************************************************************************.............***.........******.........................................*******X | 370.000 | | 50.000 | .......*X | 50.000 | | 0.000 | ***********************************************************************************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:03:40,250 | test | INFO | pool-15-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 60.000 | ........******....................................*X | 300.000 | | 240.000 | ...................................*****X | 240.000 | | 0.000 | *********************************......**.......******.....................................*X | 330.000 | | 310.000 | .............................................*****X | 310.000 | | 30.000 | ...*X | 30.000 | | 0.000 | *************************************................................................***............****X | 400.000 | | 80.000 | ...........**X | 80.000 | | 0.000 | ***********************************************.******........................***.........*****......**X | 270.000 | | 210.000 | ..............................****X | 210.000 | | 10.000 | **X | 10.000 | | 0.000 | ****************************************************************...******..............................................................................**...................*****X | 660.000 | | 260.000 | ......................................*X | 260.000 | | 120.000 | .................*****X | 120.000 | | 0.000 | ***************************************************************************************************...******........................**.*****X | 190.000 | | 150.000 | .....................*X | 150.000 | | 140.000 | ....................*****X | 140.000 | | 20.000 | ..**X | 20.000 | | 0.000 | *************************************************************************************************.******....................................***......****X | 290.000 | | 90.000 | ............**X | 90.000 | | 50.000 | ......*****............*............**.........*****X | 270.000 | | 150.000 | .....................*X | 150.000 | | 80.000 | ...........*****X | 80.000 | | 0.000 | ******************************************************************************************************************......**.******........................**......*****X | 250.000 | | 50.000 | ......*X | 50.000 | | 0.000 | **************............***...****X | 100.000 | | 20.000 | ..**X | 20.000 | | 0.000 | *******************************************************************************************************......***......*****............**X | 160.000 | | 130.000 | ..................********..*****X | 140.000 | | 20.000 | ..*X | 20.000 | | 10.000 | ******.................................................**.........*****X | 390.000 | | 70.000 | .........*X | 70.000 | | 0.000 | ******************************************************************************************************************.******..................***..****X | 140.000 | | 110.300 | ...............X | 110.300 | | 20.000 | ..**....******....................................*X | 290.000 | | 250.000 | ....................................*****X | 250.000 | | 10.000 | *X | 10.000 | | 0.000 | ************************************************************************************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:03:41,046 | test | INFO | pool-15-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...******...............................................................................**..................*****X | 71588.600 | | 62140.200 | .....................................................................................*X | 62140.200 | | 57021.100 | ..............................................................................*****X | 57021.100 | | 56892.600 | ..............................................................................**.....******.............*X | 69723.300 | | 69346.300 | ...............................................................................................*****X | 69346.300 | | 42035.300 | .........................................................**X | 42035.300 | | 34382.900 | ...............................................********.....*****..................**.*****X | 52114.400 | | 48474.400 | ..................................................................**.******..............*******X | 59300.700 | | 30644.500 | .........................................**X | 30644.500 | | 23938.500 | ................................*****...................................**..............******..........*X | 65983.200 | | 63973.100 | ........................................................................................*****X | 63973.100 | | 53925.100 | ..........................................................................*X | 53925.100 | | 50089.200 | ....................................................................*****X | 50089.200 | | 29865.800 | ........................................*X | 29865.800 | | 26826.800 | ....................................******.**..******............................**...*****X | 51100.500 | | 43710.000 | ............................................................********...............**...*****X | 57153.900 | | 52927.800 | ........................................................................*X | 52927.800 | | 50642.600 | .....................................................................****X | 50642.600 | | 42237.300 | ..........................................................**X | 42237.300 | | 40830.400 | ........................................................*****X | 40830.400 | | 31544.400 | ...........................................*X | 31544.400 | | 28391.600 | ......................................*****X | 28391.600 | | 26660.200 | ....................................**..******..............**...******.......*******X | 45553.400 | | 30873.500 | ..........................................*X | 30873.500 | | 27904.400 | .....................................******........**...******.............**.*****X | 45117.800 | | 32506.800 | ............................................*X | 32506.800 | | 28987.300 | .......................................*****X | 28987.300 | | 17603.500 | .......................*X | 17603.500 | | 15576.800 | ....................******......**...******..**X | 23613.400 | | 23543.800 | ...............................*****...........***..*****.**X | 33588.600 | | 33330.500 | .............................................****X | 33330.500 | | 24368.900 | .................................**X | 24368.900 | | 22234.500 | ..............................****X | 22234.500 | | 20777.300 | ............................**X | 20777.300 | | 20605.200 | ...........................*****.......***..******.............**...*****X | 38025.700 | | 31883.800 | ...........................................*X | 31883.800 | | 28677.800 | .......................................*****X | 28677.800 | | 16532.400 | ......................*X | 16532.400 | | 14692.800 | ...................******.********..**..******............**...*****X | 28895.400 | | 25522.200 | ..................................**X | 25522.200 | | 24155.600 | ................................****X | 24155.600 | | 20679.200 | ...........................**X | 20679.200 | | 19596.300 | ..........................*****X | 19596.300 | | 14235.400 | ..................**.******.......*X | 19641.400 | | 19275.100 | .........................******.*******X | 19855.200 | | 19374.800 | ..........................**.******.*X | 21475.500 | | 20474.500 | ...........................******.**.******..***.****X | 23784.000 | | 20627.000 | ...........................**X | 20627.000 | | 20249.900 | ...........................****X | 20249.900 | | 19841.200 | ..........................**X | 19841.200 | | 19349.600 | ..........................*****X | 19349.600 | | 17228.000 | .......................*X | 17228.000 | | 16852.000 | ......................**************...**********.*****X | 19550.500 | | 19291.600 | .........................**X | 19291.600 | | 19268.200 | .........................*****.***.*****.....********X | 24097.100 | | 20316.800 | ...........................*X | 20316.800 | | 19626.300 | ..........................*****X | 19626.300 | | 18010.000 | ........................********...*X | 20473.200 | | 20154.800 | ...........................*****X | 20154.800 | | 17553.800 | .......................********..**X | 19055.700 | | 18377.200 | ........................****X | 18377.200 | | 16485.800 | ......................********........***...****X | 24745.900 | | 24401.200 | .................................**X | 24401.200 | | 23883.600 | ................................*****X | 23883.600 | | 17384.300 | .......................*X | 17384.300 | | 15325.500 | ....................*****X | 15325.500 | | 12476.800 | ................********.........**.******...**.*****X | 22280.500 | | 18766.600 | .........................**X | 18766.600 | | 18378.000 | ........................****X | 18378.000 | | 15778.400 | .....................********..*.......*******X | 22640.500 | | 22021.000 | .............................**.******..*X | 23686.100 | | 23087.600 | ...............................*****X | 23087.600 | | 16443.300 | .....................*X | 16443.300 | | 15542.500 | ....................******......**.*****X | 20703.800 | | 17474.300 | .......................*X | 17474.300 | | 16466.300 | ......................******.......***....****X | 24396.600 | | 21332.300 | ............................**X | 21332.300 | | 19267.700 | .........................*****X | 19267.700 | | 11917.000 | ...............*X | 11917.000 | | 11211.600 | ..............******.............**..******..**.*****X | 23927.200 | | 19890.600 | ..........................*X | 19890.600 | | 18766.300 | .........................*****X | 18766.300 | | 17672.400 | .......................*X | 17672.400 | | 17170.500 | ......................*****X | 17170.500 | | 14365.100 | ...................**X | 14365.100 | | 13127.200 | .................*****..***....*****.......**X | 22597.300 | | 21405.700 | ............................*****X | 21405.700 | | 18618.700 | .........................*******X | 19311.300 | | 16024.500 | .....................*X | 16024.500 | | 14368.600 | ...................******.....**.*****X | 18834.800 | | 9128.100 | ...........*X | 9128.100 | | 8610.900 | ...........******...............***.****X | 20268.700 | | 16608.800 | ......................**X | 16608.800 | | 16476.600 | ......................****X | 16476.600 | | 16092.400 | .....................X | 16092.400 | | 14955.800 | ...................**.******......**.*****X | 20519.600 | | 19397.900 | ..........................*X | 19397.900 | | 18232.700 | ........................*****X | 18232.700 | | 12098.200 | ...............*X | 12098.200 | | 11375.200 | ..............******.***.*****........***.*****X | 18937.200 | | 17331.600 | .......................********.....**..******...*X | 24376.000 | | 24138.600 | ................................*****X | 24138.600 | | 22685.800 | ..............................*X | 22685.800 | | 22279.700 | ..............................*****X | 22279.700 | | 19485.200 | ..........................*X | 19485.200 | | 19040.000 | .........................*************X | 19055.300 | | 15358.800 | ....................**X | 15358.800 | | 15014.700 | ...................*****........********..***.*****X | 22395.300 | | 16280.900 | .....................*X | 16280.900 | | 15411.400 | ....................******......**...******..*X | 23140.500 | | 21283.100 | ............................*****X | 21283.100 | | 16604.800 | ......................**X | 16604.800 | | 15828.400 | .....................****X | 15828.400 | | 11209.500 | ..............**X | 11209.500 | | 11093.700 | ..............******...............**...*****X | 23825.400 | | 14569.800 | ...................*X | 14569.800 | | 12562.200 | ................******....**..******.....*X | 20513.100 | | 19963.600 | ..........................*****X | 19963.600 | | 13083.500 | .................**X | 13083.500 | | 12469.000 | ................*****...***.******..........**.*****X | 23149.600 | | 19215.800 | .........................*X | 19215.800 | | 18168.500 | ........................*****X | 18168.500 | | 17156.700 | ......................**.******..*X | 18777.000 | | 18626.900 | .........................*****X | 18626.900 | | 15397.200 | ....................***.*****.**X | 16563.400 | | 16064.900 | .....................*****....*.**.*****X | 20741.500 | | 17815.200 | .......................*X | 17815.200 | | 17009.900 | ......................******..**.***X | 18688.200 | +-----------+----------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:03:42,306 | test | INFO | pool-15-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 18688.200 | ...................................................................................................X | 18688.200 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:03:42,426 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ | 172.23.105.36 | kv | 51.8513789581 | 11.45 GiB | 8.56 GiB | 52.50 MiB / 3.50 GiB | 6519686 / 13054322 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 39.0172303765 | 11.45 GiB | 8.81 GiB | 24.00 MiB / 3.50 GiB | 6524746 / 13076313 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 48.4770595039 | 11.45 GiB | 8.59 GiB | 28.14 MiB / 3.50 GiB | 6541193 / 12980867 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 50.1675257732 | 11.45 GiB | 8.57 GiB | 0.0 Byte / 3.50 GiB | 6501091 / 13070775 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 35.4398059492 | 11.45 GiB | 8.77 GiB | 0.0 Byte / 3.50 GiB | 6462082 / 12902385 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+--------------------+-----------------------+ 2022-02-02 19:03:51,601 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 19:04:05,882 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:04:05,882 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:04:16,743 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:04:16,743 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:04:27,586 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:04:27,586 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:04:45,414 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 50000 | 9.77 GiB | 253.90 MiB | 335.09 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 496.68 MiB | 535.30 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 32575000 | 2.50 GiB | 1.72 GiB | 50.31 GiB | 18.5467751343 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 19:04:45,414 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2022-02-02 19:04:50,023 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during hard_failover_recovery 2022-02-02 19:04:50,023 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : hard_failover_recovery 2022-02-02 19:04:50,023 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:724] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 19:04:59,154 | test | INFO | pool-15-thread-22 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 8.03999996185 seconds to complete 2022-02-02 19:04:59,157 | test | INFO | pool-15-thread-22 | [common_lib:sleep:22] Sleep 8.03999996185 seconds. Reason: Wait after rebalance complete 2022-02-02 19:10:07,198 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 19:10:07,200 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 19:10:07,206 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 19:21:49,352 | test | INFO | pool-15-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **............................******........................................................................*X | 466.200 | | 374.400 | ...............................................................................*****X | 374.400 | | 53.600 | ..........*X | 53.600 | | 34.200 | ......******.......................**...*****X | 155.100 | | 65.400 | .............*X | 65.400 | | 53.400 | ..........******..............**.....*****X | 143.400 | | 69.300 | .............*X | 69.300 | | 21.200 | ...******..........**.......*****X | 100.600 | | 87.300 | .................*X | 87.300 | | 73.400 | ..............*X | 73.400 | | 73.400 | ..............**X | 73.400 | | 73.400 | ..............X | 73.400 | | 44.300 | ........*X | 44.300 | | 36.700 | ......******..***..*****...**X | 66.400 | | 57.900 | ...........*X | 57.900 | | 57.900 | ...........**X | 57.900 | | 57.900 | ...........X | 57.900 | | 13.000 | .*X | 13.000 | | 8.100 | ******................**...*****X | 95.900 | | 61.800 | ............*X | 61.800 | | 58.000 | ...........**X | 58.000 | | 58.000 | ...........*X | 58.000 | | 58.000 | ...........X | 58.000 | | 34.300 | ......*X | 34.300 | | 28.900 | .....******........**...*****X | 83.300 | | 80.700 | ................**X | 80.700 | | 76.000 | ...............********..*****..**X | 95.600 | | 68.000 | .............****X | 68.000 | | 41.900 | .......***....*****..**X | 67.800 | | 56.500 | ...........****X | 56.500 | | 53.700 | ..........***..****X | 61.000 | | 55.900 | ..........**X | 55.900 | | 47.000 | .........****X | 47.000 | | 28.700 | .....*********...........**...*****X | 94.700 | | 31.800 | .....*X | 31.800 | | 16.000 | ..******.....**...*****.**X | 56.800 | | 44.000 | ........*****..***...****X | 67.200 | | 64.800 | ............**X | 64.800 | | 59.200 | ...........****X | 59.200 | | 53.200 | ..........**X | 53.200 | | 45.800 | ........****X | 45.800 | | 38.100 | .......**X | 38.100 | | 35.800 | ......*****X | 35.800 | | 12.300 | .**.******.....*X | 38.500 | | 37.500 | .......******.......**..*****X | 79.400 | | 42.800 | ........*X | 42.800 | | 40.600 | .......********.******...*X | 59.800 | | 57.100 | ...........*****X | 57.100 | | 23.000 | ...**X | 23.000 | | 16.000 | ..********..******.....*X | 48.200 | | 39.400 | .......*****X | 39.400 | | 10.600 | .*X | 10.600 | | 5.400 | ******...**..*****X | 28.000 | | 7.000 | *X | 7.000 | | 2.200 | ******.........**..*****X | 53.800 | | 28.000 | .....**X | 28.000 | | 24.000 | ....****X | 24.000 | | 21.000 | ...**X | 21.000 | | 18.000 | ..*****.***.******.......**....****X | 74.800 | | 69.400 | .............**X | 69.400 | | 57.200 | ...........****X | 57.200 | | 23.000 | ...**X | 23.000 | | 16.000 | ..****X | 16.000 | | 14.100 | ..X | 14.100 | | 13.000 | .**.*****.*...*******X | 36.800 | | 12.800 | .*X | 12.800 | | 8.000 | *****.***.*************....***..****X | 42.000 | | 39.600 | .......***.****X | 42.600 | | 21.000 | ...**X | 21.000 | | 5.400 | *****..***..*****X | 24.000 | | 13.200 | .*X | 13.200 | | 8.000 | *************X | 8.000 | | 2.600 | *X | 2.600 | | 0.000 | ******.....***..*****X | 36.600 | | 34.200 | ......*X | 34.200 | | 29.400 | .....*****X | 29.400 | | 13.200 | .*X | 13.200 | | 8.000 | *****X | 8.000 | | 2.600 | **.******..**X | 16.000 | | 13.400 | .*********X | 13.400 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:21:50,187 | test | INFO | pool-15-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................******..............................................................................*X | 170.300 | | 162.200 | ..............................................................................................*****X | 162.200 | | 29.700 | ................*X | 29.700 | | 10.800 | .....******.....................................**.......*****X | 85.200 | | 37.500 | .....................*X | 37.500 | | 29.400 | ................******......*******X | 40.000 | | 37.200 | ....................**..******...............................**........*****X | 105.900 | | 55.700 | ...............................*X | 55.700 | | 45.400 | .........................******...**X | 50.700 | | 42.600 | ........................********...******..................**..........*****X | 96.000 | | 53.600 | ..............................*X | 53.600 | | 32.000 | .................*****X | 32.000 | | 26.700 | ..............**.....******...............................**..*****X | 90.500 | | 42.800 | ........................**X | 42.800 | | 32.000 | .................*****..............***....................X | 89.800 | | 89.800 | ...................................................**X | 89.800 | | 89.800 | ...................................................*.................**X | 118.900 | | 86.900 | ..................................................****X | 86.900 | | 34.100 | ...................***...*****.........**X | 55.200 | | 41.700 | .......................****X | 41.700 | | 40.800 | ......................***.............X | 61.900 | | 61.900 | ...................................****..................**X | 93.100 | | 93.000 | .....................................................****X | 93.000 | | 44.300 | .........................**X | 44.300 | | 23.300 | ............******..............**....********....*****X | 60.000 | | 49.400 | ............................*X | 49.400 | | 44.000 | ........................*****X | 44.000 | | 36.300 | ....................*X | 36.300 | | 34.000 | ..................******..............................**..............****X | 108.900 | | 64.000 | ....................................**X | 64.000 | | 33.100 | ..................*****...***......******.................**......*****X | 88.000 | | 58.900 | .................................*X | 58.900 | | 46.400 | ..........................*****X | 46.400 | | 23.000 | ............*X | 23.000 | | 21.400 | ...........******........*X | 34.500 | | 28.700 | ...............******....***................******...........................................*X | 135.700 | | 114.600 | ..................................................................*****X | 114.600 | | 33.300 | ..................*X | 33.300 | | 33.200 | ..................******....................**...*****X | 72.800 | | 48.400 | ...........................*X | 48.400 | | 40.000 | ......................******........**.....*****X | 62.000 | | 30.400 | ................*X | 30.400 | | 18.000 | .........*****X | 18.000 | | 10.200 | ....********..**X | 12.600 | | 7.800 | ...*****..............***..........******...........*X | 67.000 | | 61.200 | ..................................*****X | 61.200 | | 27.000 | ..............*X | 27.000 | | 20.000 | ..........******.......**....******............***..****X | 61.400 | | 35.000 | ...................**X | 35.000 | | 28.000 | ...............****X | 28.000 | | 27.300 | ...............X | 27.300 | | 25.600 | ..............*******.......................***........****X | 80.000 | | 66.900 | ......................................**X | 66.900 | | 62.000 | ...................................****X | 62.000 | | 40.500 | ......................X | 40.500 | | 30.200 | ................*X | 30.200 | | 20.600 | ...........******...............**....*****X | 54.000 | | 19.800 | ..........*X | 19.800 | | 10.600 | .....*****......**X | 21.400 | | 18.800 | ..........****X | 18.800 | | 8.000 | ...***..*****...............***..........*****...........................***..*****X | 103.000 | | 47.200 | ..........................*X | 47.200 | | 29.000 | ................******......**.*****X | 41.800 | | 15.400 | ........**X | 15.400 | | 8.000 | ...*****...***........*****................**X | 53.400 | | 45.600 | .........................*****X | 45.600 | | 18.600 | .........********....*X | 24.000 | | 21.400 | ...........******.................**...........*****X | 68.600 | | 33.800 | ..................**X | 33.800 | | 15.600 | ........*****.***.....******..............*X | 50.800 | | 45.600 | .........................**X | 45.600 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:21:51,045 | test | INFO | pool-15-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....******...........................................................................**....................*****X | 60882.100 | | 38936.400 | ..............................................................*X | 38936.400 | | 32388.900 | ....................................................******.............**....*****X | 42932.800 | | 41089.200 | ..................................................................**..******.................**.*****X | 53070.300 | | 41827.500 | ...................................................................*X | 41827.500 | | 38987.300 | ...............................................................*****X | 38987.300 | | 38404.200 | ..............................................................**.*****X | 39064.700 | | 38606.100 | ..............................................................*X | 38606.100 | | 38365.900 | ..............................................................******.....*X | 41610.100 | | 41326.200 | ..................................................................*****X | 41326.200 | | 39504.300 | ...............................................................**.*****X | 39994.100 | | 35494.400 | .........................................................*X | 35494.400 | | 33291.500 | .....................................................******.......***...****X | 39104.800 | | 35536.900 | .........................................................***.*****..............*******X | 44723.700 | | 33849.600 | ......................................................**X | 33849.600 | | 31633.700 | ..................................................******..................**.....*****X | 45380.800 | | 36189.500 | ..........................................................*X | 36189.500 | | 33482.800 | .....................................................******.......**...*****X | 39361.400 | | 36631.100 | ...........................................................*X | 36631.100 | | 32916.700 | .....................................................********.....******..................**....******...........**..*****X | 57558.300 | | 43475.900 | ......................................................................*X | 43475.900 | | 39511.200 | ...............................................................*****X | 39511.200 | | 37599.600 | ............................................................**X | 37599.600 | | 35270.200 | ........................................................****X | 35270.200 | | 32427.100 | ....................................................***.*****....********X | 35871.600 | | 33490.400 | ......................................................*******...........***.****X | 41190.500 | | 35978.900 | ..........................................................**X | 35978.900 | | 33888.600 | ......................................................****X | 33888.600 | | 30807.800 | .................................................***.****X | 31095.900 | | 30369.700 | ................................................**X | 30369.700 | | 30160.000 | ................................................*****..*******X | 31502.100 | | 31265.100 | ..................................................**X | 31265.100 | | 31014.400 | .................................................*****X | 31014.400 | | 30507.500 | .................................................**.******..*X | 32545.200 | | 32094.800 | ...................................................*****X | 32094.800 | | 29713.800 | ...............................................*X | 29713.800 | | 28941.600 | ..............................................******..**.******.....*X | 33569.700 | | 33505.900 | ......................................................*****X | 33505.900 | | 28777.500 | ..............................................*X | 28777.500 | | 27485.100 | ............................................******...**.*****X | 29930.000 | | 26216.000 | ..........................................*X | 26216.000 | | 24782.100 | .......................................******...**.******....********..***..****X | 32152.500 | | 28605.500 | .............................................**X | 28605.500 | | 26666.400 | ..........................................*****X | 26666.400 | | 26152.600 | .........................................**.******.....**..*****X | 30589.700 | | 28332.800 | .............................................*X | 28332.800 | | 27967.500 | ............................................*****X | 27967.500 | | 26706.500 | ..........................................*X | 26706.500 | | 25806.100 | .........................................******....**.******.*X | 29292.200 | | 28895.100 | ..............................................*****X | 28895.100 | | 23979.800 | ......................................*X | 23979.800 | | 22499.200 | ...................................*****X | 22499.200 | | 21375.000 | ..................................*******X | 21742.500 | | 20893.700 | .................................**X | 20893.700 | | 20205.300 | ................................*****....***.*****.....***....****X | 28624.300 | | 27056.100 | ...........................................**X | 27056.100 | | 24927.400 | .......................................****X | 24927.400 | | 24194.400 | ......................................**X | 24194.400 | | 24087.500 | ......................................****X | 24087.500 | | 23989.900 | ......................................***.****X | 24469.900 | | 22393.300 | ...................................**X | 22393.300 | | 21633.000 | ..................................*****X | 21633.000 | | 21459.200 | ..................................********.*******X | 22071.500 | | 20729.500 | .................................*X | 20729.500 | | 20214.300 | ................................******.....**..*****X | 24656.400 | | 20615.100 | ................................**X | 20615.100 | | 18040.800 | ............................******...**...*****X | 21915.500 | | 21381.000 | ..................................*X | 21381.000 | | 21025.500 | .................................******..**..******..*X | 24804.700 | | 24417.600 | .......................................***X | 24417.600 | +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:21:52,312 | test | INFO | pool-15-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 13.400 | ...................................................................................................X | 13.400 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 19:21:52,326 | test | INFO | pool-15-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 45.600 | ...................................................................................................X | 45.600 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 19:21:52,342 | test | INFO | pool-15-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 24417.600 | ...................................................................................................X | 24417.600 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:21:52,459 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 19:21:52,480 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 19:22:03,243 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait for rebalance to start 2022-02-02 19:22:13,243 | test | INFO | MainThread | [collections_rebalance:compact_all_buckets:241] Starting compaction for each bucket 2022-02-02 19:22:13,272 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveAdded 2022-02-02 19:22:13,299 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveAdded 2022-02-02 19:22:13,322 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveAdded 2022-02-02 19:22:13,364 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 19:22:13,369 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 19:22:13,479 | test | INFO | pool-15-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 4.39532839382 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 19.7053638882 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 7.16976393772 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.375657400451 | Cluster node | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 9.04977375566 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 19:22:13,499 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 19:22:23,526 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 19:22:33,563 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 6.14 2022-02-02 19:22:43,605 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 15.99 2022-02-02 19:22:45,444 | test | INFO | pool-15-thread-11 | [task:call:6002] bucket1 compaction done: 100% 2022-02-02 19:22:53,638 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 27.39 2022-02-02 19:23:03,676 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 34.09 2022-02-02 19:23:13,720 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 43.63 2022-02-02 19:23:23,760 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 54.72 2022-02-02 19:23:33,798 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 59.44 2022-02-02 19:23:43,841 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 63.56 2022-02-02 19:23:52,632 | test | INFO | pool-15-thread-2 | [task:call:6002] bucket2 compaction done: 100% 2022-02-02 19:23:53,884 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 66.9 2022-02-02 19:24:03,921 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 67.49 2022-02-02 19:24:13,963 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.25 2022-02-02 19:24:24,009 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 68.72 2022-02-02 19:24:34,045 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 69.19 2022-02-02 19:24:44,082 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 69.89 2022-02-02 19:24:54,118 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.36 2022-02-02 19:25:04,164 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 70.68 2022-02-02 19:25:14,201 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 71.21 2022-02-02 19:25:24,239 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 71.8 2022-02-02 19:25:34,288 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.39 2022-02-02 19:25:44,323 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 72.8 2022-02-02 19:25:54,368 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 73.27 2022-02-02 19:26:04,421 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 73.74 2022-02-02 19:26:14,457 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 74.27 2022-02-02 19:26:24,494 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 74.71 2022-02-02 19:26:34,540 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 75.23 2022-02-02 19:26:44,581 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 76.05 2022-02-02 19:26:54,618 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 76.28 2022-02-02 19:27:04,654 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 77.04 2022-02-02 19:27:14,697 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 77.75 2022-02-02 19:27:24,740 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 78.49 2022-02-02 19:27:34,773 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 79.16 2022-02-02 19:27:44,812 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 80.02 2022-02-02 19:27:54,848 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 80.81 2022-02-02 19:28:04,885 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 81.6 2022-02-02 19:28:14,921 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 82.43 2022-02-02 19:28:24,967 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.1 2022-02-02 19:28:35,012 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 83.62 2022-02-02 19:28:45,059 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 84.35 2022-02-02 19:28:55,098 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 84.92 2022-02-02 19:29:05,132 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 85.75 2022-02-02 19:29:15,167 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.37 2022-02-02 19:29:25,209 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 86.84 2022-02-02 19:29:35,246 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 87.41 2022-02-02 19:29:45,280 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.14 2022-02-02 19:29:55,315 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 88.61 2022-02-02 19:30:05,351 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.08 2022-02-02 19:30:15,388 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 89.65 2022-02-02 19:30:25,427 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 90.16 2022-02-02 19:30:35,467 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 90.68 2022-02-02 19:30:45,506 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.1 2022-02-02 19:30:55,542 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 91.72 2022-02-02 19:31:05,575 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 92.45 2022-02-02 19:31:12,996 | test | ERROR | pool-15-thread-29 | [task:execute_tasks:3940] Failed to load 35 docs from 26060 to 51077 of thread_name LoadDocs_MutateDocsFromSpecTask_1643858538.59_default__default_RRKBLiFQuGTyACjtDuKVqegK2D%NbMZF8eh1sTLZSyDKFgeOb4GLN0aCtKp%vQ9HByk36ANeimElcjS51k52KGTq2p9XRi0tgsav8z9lP5eRSgOOf3qIb9m6JnOZ25Kkp3IYHEHOVCZ9rejp5awXpJfPREhIRGSNFbwX_GawP-QwLSKotVvsNYc3ZU-n2ABXnOwRVcATSxQ0opXR5tCrOcvjy0mj%nH5fyxDDsVM9USz3-30-995000_ttl=0_delete__26060_51077_1643858539.1 2022-02-02 19:31:15,612 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.17 2022-02-02 19:31:25,645 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 93.69 2022-02-02 19:31:35,680 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 94.16 2022-02-02 19:31:45,724 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 94.78 2022-02-02 19:31:55,762 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 95.3 2022-02-02 19:32:05,796 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 95.83 2022-02-02 19:32:15,838 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.34 2022-02-02 19:32:25,885 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 96.81 2022-02-02 19:32:35,928 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.28 2022-02-02 19:32:45,966 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.6 2022-02-02 19:32:56,003 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 97.97 2022-02-02 19:33:06,036 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.29 2022-02-02 19:33:16,072 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 98.71 2022-02-02 19:33:26,105 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.09 2022-02-02 19:33:36,138 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.36 2022-02-02 19:33:46,171 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: running, progress: 99.84 2022-02-02 19:34:01,224 | test | INFO | pool-15-thread-24 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 19:34:01,244 | test | INFO | pool-15-thread-24 | [task:check:468] Rebalance completed with progress: 100% in 707.765999794 sec 2022-02-02 19:41:52,750 | test | INFO | pool-15-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.....................................................................***...............................****X | 321.200 | | 118.500 | ...................................**X | 118.500 | | 30.400 | ........*****............***.....****X | 85.600 | | 26.000 | .......**X | 26.000 | | 0.000 | ******.............**.....*****X | 60.800 | | 29.800 | ........*X | 29.800 | | 17.100 | ....******..**X | 23.400 | | 21.100 | .....*****X | 21.100 | | 9.300 | .*X | 9.300 | | 7.600 | .******....**.******...............**...*****X | 81.900 | | 34.700 | .........**X | 34.700 | | 24.500 | ......******..**.*****X | 32.400 | | 25.000 | ......*X | 25.000 | | 21.100 | .....*****X | 21.100 | | 3.400 | **X | 3.400 | | 1.700 | *****.............***...********..*****X | 58.300 | | 46.500 | .............*X | 46.500 | | 34.600 | .........*****X | 34.600 | | 11.000 | ..**X | 11.000 | | 7.600 | .*****....***.******...**.******..**..*****X | 49.000 | | 43.100 | ............*X | 43.100 | | 36.300 | ..........*****X | 36.300 | | 12.700 | ..**X | 12.700 | | 7.600 | .******..**.*****X | 17.200 | | 9.800 | ..*X | 9.800 | | 5.900 | ******.....**..******........***..*****.....*******X | 76.300 | | 15.800 | ...**X | 15.800 | | 0.000 | ******..**..******....................**....*****X | 91.800 | | 29.800 | ........*X | 29.800 | | 15.200 | ...*****X | 15.200 | | 9.300 | .********........***....*****.............**X | 89.100 | | 80.400 | ........................*****X | 80.400 | | 9.500 | .*X | 9.500 | | 3.400 | ******.....*X | 21.100 | | 19.400 | .....********.******....**...******.............*X | 87.600 | | 83.600 | .........................*****X | 83.600 | | 45.500 | .............**X | 45.500 | | 42.000 | ............****X | 42.000 | | 28.700 | .......**X | 28.700 | | 24.800 | ......*****X | 24.800 | | 13.000 | ...*X | 13.000 | | 9.600 | .*****X | 9.600 | | 8.100 | .**.******........*******X | 38.100 | | 11.600 | ..*X | 11.600 | | 9.300 | .*******X | 9.300 | | 5.900 | ******...**..******.*X | 25.300 | | 24.800 | ......*****X | 24.800 | | 11.500 | ..*X | 11.500 | | 8.000 | .******..***.*************..........***..******..**.*****X | 65.200 | | 22.300 | .....*X | 22.300 | | 9.600 | .******...*******X | 19.200 | | 4.400 | ********....***.*****....**X | 34.200 | | 32.500 | .........*****......***..*****X | 61.000 | | 12.200 | ..*X | 12.200 | | 0.000 | **************......**..*****X | 26.800 | | 6.100 | **X | 6.100 | | 0.000 | *****.............***....****X | 55.600 | | 26.000 | .......**X | 26.000 | | 17.200 | ....*****...***.*****X | 31.000 | | 23.600 | ......*X | 23.600 | | 17.000 | ....*****X | 17.000 | | 2.200 | *X | 2.200 | | 0.000 | ******..********......***....*****...........*******X | 74.600 | | 19.200 | ....**X | 19.200 | | 7.600 | .********....*****...............**X | 68.600 | | 58.800 | .................****X | 58.800 | | 17.600 | ....***.******.....*X | 36.000 | | 32.400 | .........******...............**......*****X | 101.200 | | 48.400 | ..............*X | 48.400 | | 34.200 | .........*******X | 34.200 | | 30.600 | ........*****X | 30.600 | | 26.400 | .......********...***...****X | 47.200 | | 39.800 | ...........**X | 39.800 | | 23.200 | ......****X | 23.200 | | 0.000 | ***.*****............***.****X | 47.400 | | 26.800 | .......**X | 26.800 | | 25.000 | ......******.....*X | 39.800 | | 36.200 | ..........*****X | 36.200 | | 9.800 | ..*X | 9.800 | | 7.600 | .**************..**...******.........**X | 53.400 | | 49.400 | ..............****X | 49.400 | | 18.800 | ....**X | 18.800 | | 15.200 | ...*****X | 15.200 | | 9.400 | .*X | 9.400 | | 5.800 | *****X | 5.800 | | 0.000 | ********.....**.*****X | 21.000 | | 3.600 | ***.*****...........**X | 42.000 | | 40.200 | ...........*****X | 40.200 | | 11.200 | ..*X | 11.200 | | 5.800 | *****X | 5.800 | | 0.000 | ********....*******X | 13.400 | | 1.800 | **X | 1.800 | | 0.000 | **********************...**.*****X | 15.200 | | 3.600 | **X | 3.600 | | 0.000 | *************...***.*****X | 15.200 | | 9.400 | .*X | 9.400 | | 7.600 | .******...**..*****X | 22.800 | | 11.200 | ..*X | 11.200 | | 9.400 | .******..**X | 15.200 | | 13.400 | ...****X | 13.400 | | 1.800 | **X | 1.800 | | 0.000 | ******************************.***.*****X | 7.600 | | 1.800 | *X | 1.800 | | 0.000 | ****X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:41:53,447 | test | INFO | pool-15-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.....................................................................***...............................****X | 222.400 | | 89.900 | .......................................**X | 89.900 | | 32.400 | .............*****.........*******X | 53.200 | | 11.900 | ....**X | 11.900 | | 0.000 | ******..........**...******.............**......*****X | 71.800 | | 42.300 | ..................*X | 42.300 | | 27.000 | ...........*********...*******X | 33.800 | | 28.700 | ...........*****X | 28.700 | | 16.900 | ......*X | 16.900 | | 13.500 | .....******.....**..******..............**.....*****X | 72.100 | | 36.700 | ...............**X | 36.700 | | 23.100 | .........*****X | 23.100 | | 15.700 | ......*X | 15.700 | | 15.200 | .....******...........**...*****X | 45.600 | | 10.200 | ...*X | 10.200 | | 1.800 | ******...........***....******...................*******X | 76.800 | | 17.800 | .......*X | 17.800 | | 7.600 | ..******...*******X | 15.200 | | 3.400 | **X | 3.400 | | 0.000 | ******..**.******...**..******...*X | 24.500 | | 22.800 | .........******.....***.*****X | 36.300 | | 30.400 | ............*X | 30.400 | | 28.700 | ...........*****X | 28.700 | | 22.800 | .........**.******.........**....******......**X | 68.200 | | 62.200 | ..........................****X | 62.200 | | 33.100 | .............**X | 33.100 | | 29.100 | ............*****..*...**.********.******.....**....****X | 65.200 | | 65.200 | ............................X | 65.200 | | 63.900 | ...........................*X | 63.900 | | 49.400 | .....................***X | 49.400 | | 49.400 | .....................*X | 49.400 | | 5.100 | .*X | 5.100 | | 2.200 | ******......*X | 15.500 | | 15.000 | .....*****X | 15.000 | | 7.600 | ..********...........***....*****.........**X | 60.600 | | 59.400 | .........................*****X | 59.400 | | 31.400 | .............*X | 31.400 | | 26.800 | ...........****X | 26.800 | | 17.900 | .......**X | 17.900 | | 16.900 | ......******......*******X | 30.700 | | 23.200 | .........*X | 23.200 | | 22.800 | .........*****X | 22.800 | | 5.200 | .*X | 5.200 | | 2.200 | ******.......*******X | 17.000 | | 15.500 | .....**.*****X | 17.200 | | 11.300 | ....*X | 11.300 | | 9.600 | ...******..**...******..**X | 25.000 | | 22.700 | .........******...*******X | 30.700 | | 18.900 | .......*X | 18.900 | | 17.200 | ......******...*******X | 23.100 | | 18.700 | .......***************X | 19.200 | | 10.200 | ...**X | 10.200 | | 9.300 | ...*****.......*..**...****X | 36.600 | | 30.800 | ............**X | 30.800 | | 24.600 | ..........********.*******X | 27.300 | | 26.800 | ...........******......**.*****X | 43.800 | | 29.100 | ............*X | 29.100 | | 29.000 | ............*****X | 29.000 | | 21.500 | ........********..........**X | 43.800 | | 39.400 | ................****X | 39.400 | | 23.600 | .........X | 23.600 | | 9.900 | ...*X | 9.900 | | 7.600 | ..*****X | 7.600 | | 1.700 | *X | 1.700 | | 0.000 | **************..**.******......***.....*****................**X | 68.600 | | 66.100 | ............................*****X | 66.100 | | 62.800 | ...........................**.****X | 65.400 | | 46.600 | ...................**X | 46.600 | | 46.400 | ...................*****..............**X | 75.700 | | 73.800 | ................................****X | 73.800 | | 32.100 | .............**X | 32.100 | | 26.200 | ..........*****X | 26.200 | | 14.100 | .....**..******...............*X | 51.800 | | 45.200 | ...................*****X | 45.200 | | 17.200 | ......********..**.******..**.*****X | 30.400 | | 12.900 | ....**X | 12.900 | | 12.000 | ....*****...............***...****X | 51.200 | | 47.000 | ....................**X | 47.000 | | 42.000 | .................*****..***......*****X | 59.600 | | 46.400 | ...................*X | 46.400 | | 26.400 | ..........******...**.......******..*X | 51.900 | | 41.800 | .................*****X | 41.800 | | 5.400 | .*X | 5.400 | | 1.800 | ******...........***..****X | 30.600 | | 5.800 | .***.*****.................**X | 45.600 | | 43.800 | ..................*****X | 43.800 | | 24.800 | ..........*******X | 24.800 | | 11.600 | ....*X | 11.600 | | 7.600 | ..*******X | 7.600 | | 5.800 | .*********...*****...........********X | 36.400 | | 17.400 | ......*X | 17.400 | | 15.200 | .....*****X | 15.200 | | 11.000 | ...**.******.......**..*****X | 32.400 | | 13.400 | .....**X | 13.400 | | 5.800 | .****X | 5.800 | | 0.000 | *********.....**..*****X | 17.400 | | 13.200 | ....*X | 13.200 | | 7.400 | ..*****X | 7.400 | | 5.800 | .**..******...*X | 17.200 | | 15.000 | .....*****X | 15.000 | | 13.400 | .....*******X | 15.200 | | 9.400 | ...**X | 9.400 | | 7.600 | ..******...**..******...*X | 24.800 | | 23.000 | .........******....**....******...**X | 47.400 | | 42.000 | .................****X | 42.000 | | 7.200 | ..**X | 7.200 | | 1.800 | ******......*******X | 13.400 | | 7.600 | ..***********.*******X | 9.400 | | 5.800 | .*****X | 5.800 | | 0.000 | ********.......**...*****X | 22.800 | | 5.400 | .*X | 5.400 | | 0.000 | *****X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:41:53,614 | test | INFO | pool-15-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***................***.........*****............********.....***...****X | 15691.200 | | 13539.100 | ......................................**X | 13539.100 | | 12533.300 | ...................................*****X | 12533.300 | | 11513.500 | ................................**.******..............**...*****X | 17662.900 | | 15102.100 | ..........................................*X | 15102.100 | | 14149.600 | ........................................*****X | 14149.600 | | 12474.000 | ...................................**X | 12474.000 | | 12188.700 | ..................................*****..***.******..**.******.....**...******...*******X | 17974.400 | | 16232.400 | ..............................................**X | 16232.400 | | 15492.800 | ............................................*****.*********....**...******.......*X | 20655.900 | | 20629.600 | ..........................................................******.********....**..*****X | 22869.200 | | 21557.000 | .............................................................**X | 21557.000 | | 20338.800 | ..........................................................*****.***.....*****.............**X | 26891.800 | | 25823.800 | ..........................................................................****X | 25823.800 | | 18241.900 | ....................................................**X | 18241.900 | | 17324.100 | .................................................******...........**..*****X | 21830.600 | | 21266.600 | ............................................................*X | 21266.600 | | 20335.700 | ..........................................................*****X | 20335.700 | | 17716.400 | ..................................................********....***.*****.....**X | 21281.500 | | 21156.400 | ............................................................*****....................***...*****X | 28991.000 | | 22531.100 | ................................................................*X | 22531.100 | | 21213.200 | ............................................................*****X | 21213.200 | | 21002.600 | ............................................................*X | 21002.600 | | 20621.300 | ..........................................................*****.X | 20779.700 | | 20657.400 | ...........................................................*******....*....**....*****...*...*X | 26860.400 | | 26038.200 | ..........................................................................*****X | 26038.200 | | 19309.400 | .......................................................*X | 19309.400 | | 18186.100 | ...................................................*****X | 18186.100 | | 17151.500 | ................................................*X | 17151.500 | | 16730.900 | ...............................................******....*******X | 18180.200 | | 17437.400 | .................................................**.******......**..******...............................**.......*****X | 33651.300 | | 25300.200 | ........................................................................*X | 25300.200 | | 20698.600 | ...........................................................*****X | 20698.600 | | 16693.500 | ...............................................**.******....*X | 18403.400 | | 18312.900 | ....................................................*****X | 18312.900 | | 10855.600 | ..............................*X | 10855.600 | | 9610.400 | ..........................******.........**.******..**..*****X | 14163.400 | | 13705.100 | ......................................**.******..............**.*****X | 18938.600 | | 17899.500 | ...................................................**..*****X | 18622.800 | | 17555.900 | ..................................................*X | 17555.900 | | 16800.200 | ...............................................*****X | 16800.200 | | 11934.800 | .................................**...******..................*X | 19178.300 | | 18200.200 | ...................................................*****X | 18200.200 | | 16496.000 | ..............................................**..*****X | 17090.700 | | 16925.400 | ................................................**X | 16925.400 | | 16046.300 | .............................................****X | 16046.300 | | 14737.800 | .........................................***...*****.......**X | 18204.800 | | 18058.500 | ...................................................****X | 18058.500 | | 16970.200 | ................................................X | 16970.200 | | 16529.900 | ...............................................**.****X | 16916.200 | | 14563.100 | .........................................*X | 14563.100 | | 13274.400 | .....................................******...............***...****X | 19291.500 | | 15802.600 | ............................................***.****X | 15880.000 | | 14961.800 | ..........................................**X | 14961.800 | | 14528.000 | .........................................*****........***...*****.**X | 18801.100 | | 18437.200 | ....................................................*****....*******X | 19795.400 | | 13680.100 | ......................................**X | 13680.100 | | 12400.500 | ...................................*****.....***..*****.......***...*****..*******X | 19254.300 | | 19080.800 | ......................................................**X | 19080.800 | | 18181.300 | ...................................................*****X | 18181.300 | | 15409.400 | ...........................................**..******.........................**.......*****...................***..****X | 34126.700 | | 32429.300 | .............................................................................................**X | 32429.300 | | 32267.900 | ............................................................................................****X | 32267.900 | | 25987.600 | ..........................................................................*X | 25987.600 | | 24518.700 | ......................................................................******......**...******..................**..*****X | 34402.600 | | 29709.400 | .....................................................................................*X | 29709.400 | | 28797.000 | ..................................................................................*****X | 28797.000 | | 23906.200 | ....................................................................**X | 23906.200 | | 22725.900 | .................................................................*****...***..*****..*******X | 25421.700 | | 23006.600 | .................................................................**X | 23006.600 | | 22136.900 | ...............................................................*****....***...*****........**X | 27422.400 | | 27404.200 | ..............................................................................****X | 27404.200 | | 22055.400 | ...............................................................**X | 22055.400 | | 21333.400 | .............................................................*****........................***....****X | 31245.400 | | 20591.100 | ..........................................................**X | 20591.100 | | 17850.900 | ..................................................****X | 17850.900 | | 16483.800 | ..............................................***...*****...........**X | 21061.300 | | 21059.500 | ............................................................*****X | 21059.500 | | 20003.400 | .........................................................*X | 20003.400 | | 19479.700 | .......................................................******.**.*****X | 20036.200 | | 18666.800 | .....................................................*********X | 18736.600 | | 18009.000 | ...................................................*****X | 18009.000 | | 16535.200 | ...............................................**.******.......*X | 19374.400 | | 18765.400 | .....................................................*****X | 18765.400 | | 14123.600 | ........................................*X | 14123.600 | | 13694.900 | ......................................******......********..***.******.....*X | 18273.500 | | 18009.400 | ...................................................*****X | 18009.400 | | 17061.200 | ................................................**.*****X | 17387.900 | | 15558.600 | ............................................*X | 15558.600 | | 14916.000 | ..........................................*****X | 14916.000 | | 14705.100 | .........................................*X | 14705.100 | | 14596.700 | .........................................******..*******X | 15442.400 | | 14439.600 | ........................................*X | 14439.600 | | 13886.700 | .......................................*****X | 13886.700 | | 12100.600 | ..................................********........*******X | 14901.600 | | 14272.600 | ........................................***.*****....**X | 16126.700 | | 15725.300 | ............................................****X | 15725.300 | | 13117.900 | .....................................********....**X | 14482.100 | | 14223.100 | ........................................*****X | 14223.100 | | 11793.200 | .................................*X | 11793.200 | | 11552.400 | ................................******........**..*******X | 14968.500 | | 14520.000 | .........................................*****X | 14520.000 | | 13995.500 | .......................................**X | 13995.500 | | 13808.300 | .......................................****X | 13808.300 | | 10225.500 | ............................**X | 10225.500 | | 9727.700 | ...........................******......**..*****X | 12525.600 | | 12314.000 | ..................................*X | 12314.000 | | 12057.800 | ..................................******.**.******................**...*****X | 19546.300 | +-----------+---------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:41:54,917 | test | INFO | pool-15-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 13475.600 | ...................................................................................................X | 13475.600 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:41:55,066 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.106.177', '172.23.105.206']} 2022-02-02 19:42:01,263 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 19:42:15,538 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:42:15,538 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:42:26,437 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:42:26,437 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:42:37,298 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:42:37,298 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:42:55,114 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 46000 | 9.77 GiB | 241.80 MiB | 349.10 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 4.88 GiB | 409.48 MiB | 450.08 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 30021250 | 2.50 GiB | 1.72 GiB | 49.95 GiB | 32.1968339093 | +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+ 2022-02-02 19:42:55,595 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 19:42:55,595 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 19:43:02,270 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 19:43:02,270 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 19:43:08,950 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 39% 2022-02-02 19:43:08,950 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 0% 2022-02-02 19:44:04,361 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 19:44:04,803 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 19:44:04,803 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 19:44:34,951 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 19:44:35,365 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 19:44:35,365 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 19:45:05,506 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 19:45:05,924 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 19:45:05,924 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 19:48:07,365 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1666] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7af74b65cbf5ce0ba460f5fdd9da5c1f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=585afccc203f342a4edf45baa984bc63', u'status': u'notRunning'} - rebalance failed 2022-02-02 19:48:07,391 | test | INFO | MainThread | [rest_client:print_UI_logs:2829] Latest logs from UI on 172.23.105.164: 2022-02-02 19:48:07,391 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643860086134L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:48:06.134Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2","bucket1","default"]}}}]}.\nRebalance Operation Id = 3e946ce66fd00e86af3e238bfd5bd4f4'} 2022-02-02 19:48:07,392 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643860086133L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:48:06.133Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.105.164\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["bucket2",\n "bucket1",\n "default"]}}}]\n'} 2022-02-02 19:48:07,392 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.106.177', u'tstamp': 1643859979811L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:46:19.811Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 19:48:07,392 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643859906130L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:45:06.130Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.164'], EjectNodes = ['ns_1@172.23.105.36',\n 'ns_1@172.23.105.33',\n 'ns_1@172.23.105.206',\n 'ns_1@172.23.106.177'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 3e946ce66fd00e86af3e238bfd5bd4f4"} 2022-02-02 19:48:07,392 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643859906122L, u'shortText': u'client-side error report', u'serverTime': u'2022-02-02T19:45:06.122Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.105.164\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.105.36\', u\'ns_1@172.23.105.33\', u\'ns_1@172.23.105.206\', u\'ns_1@172.23.106.177\']'} 2022-02-02 19:48:07,392 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.33', u'tstamp': 1643859845024L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:44:05.024Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 19:48:07,392 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643859843777L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:44:03.777Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 19:48:07,394 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.36', u'tstamp': 1643859838709L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:43:58.709Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 19:48:07,394 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.105.206', u'tstamp': 1643859838598L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:43:58.598Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_writer_threads">>]'} 2022-02-02 19:48:07,394 | test | ERROR | MainThread | [rest_client:print_UI_logs:2831] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.164', u'tstamp': 1643859814453L, u'shortText': u'message', u'serverTime': u'2022-02-02T19:43:34.453Z', u'text': u'Shutting down bucket "bucket1" on \'ns_1@172.23.105.164\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 636, in tearDownEverything 2022-02-02 19:48:07,394 | test | WARNING | MainThread | [basetestcase:tearDownEverything:641] Killing memcached due to Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7af74b65cbf5ce0ba460f5fdd9da5c1f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=585afccc203f342a4edf45baa984bc63', u'status': u'notRunning'} - rebalance failed self.cluster_util.cluster_cleanup(cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 169, in cluster_cleanup self.cleanup_cluster(cluster, master=cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 209, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 143, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1569, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1706, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1668, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'7af74b65cbf5ce0ba460f5fdd9da5c1f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=585afccc203f342a4edf45baa984bc63', u'status': u'notRunning'} - rebalance failed ok ---------------------------------------------------------------------- Ran 1 test in 4273.675s OK During the test, Remote Connections: 259, Disconnections: 259 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 12 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_12 Logs will be stored at /data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_13 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-temp_rebalance_magma.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,disk_optimized_thread_settings=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability,durability=MAJORITY,bucket_spec=magma_dgm.40_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_rebalance_out_P0_set0' Test Input params: {'doc_size': '512', 'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-temp_rebalance_magma', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/win10-bucket-ops-temp_rebalance_magma.ini', 'get-cbcollect-info': 'True', 'durability': 'MAJORITY', 'conf_file': 'conf/magma/dgm_collections_failover.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'dgm_collections_failover', 'rerun': 'False', 'num_nodes': 7, 'logs_folder': '/data/workspace/temp_rebalance_magma/logs/testrunner-22-Feb-01_23-35-30/test_13', 'nodes_init': '5', 'GROUP': 'graceful_failover_and_rebalance_out_P0_set0', 'disk_optimized_thread_settings': 'True', 'bucket_spec': 'magma_dgm.40_percent_dgm.5_node_2_replica_magma_512', 'case_number': 13, 'randomize_value': 'True', 'override_spec_params': 'durability'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2022-02-02 19:48:13,108 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionsRebalance setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:48:13,506 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:50:22,571 | test | INFO | MainThread | [rest_client:monitorRebalance:1599] Rebalance done. Taken 128.781000137 seconds to complete 2022-02-02 19:50:22,579 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 19:50:27,670 | test | INFO | MainThread | [basetestcase:initialize_cluster:432] Initializing cluster : C1 Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xccb53d type=client open_count=1 channel=[id: 0x786bfeab, 0.0.0.0/0.0.0.0:36124] timeout=300.0>) 2022-02-02 19:50:31,584 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xccb53e type=client open_count=1 channel=[id: 0x9cd1235f, 0.0.0.0/0.0.0.0:43976] timeout=300.0>) 2022-02-02 19:50:33,595 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xccb53f type=client open_count=1 channel=[id: 0x6c1463c5, 0.0.0.0/0.0.0.0:56495] timeout=300.0>) 2022-02-02 19:50:35,605 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.164:8091 during connect (<_realsocket at 0xccb540 type=client open_count=1 channel=[id: 0xab0d40b0, 0.0.0.0/0.0.0.0:33879] timeout=300.0>) 2022-02-02 19:50:37,615 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.164:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xccb541 type=client open_count=1 channel=[id: 0x425b1c8e, 0.0.0.0/0.0.0.0:54542] timeout=300.0>) 2022-02-02 19:50:43,013 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xccb542 type=client open_count=1 channel=[id: 0x74406b64, 0.0.0.0/0.0.0.0:46858] timeout=300.0>) 2022-02-02 19:50:45,023 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xccb543 type=client open_count=1 channel=[id: 0xbe01799c, 0.0.0.0/0.0.0.0:44332] timeout=300.0>) 2022-02-02 19:50:47,033 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.206:8091 during connect (<_realsocket at 0xccb544 type=client open_count=1 channel=[id: 0x50ef1bde, 0.0.0.0/0.0.0.0:44465] timeout=300.0>) 2022-02-02 19:50:49,043 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xccb545 type=client open_count=1 channel=[id: 0x4f2aeb4c, 0.0.0.0/0.0.0.0:50116] timeout=300.0>) 2022-02-02 19:50:54,822 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xccb546 type=client open_count=1 channel=[id: 0x41cd6bbf, 0.0.0.0/0.0.0.0:52832] timeout=300.0>) 2022-02-02 19:50:56,835 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xccb547 type=client open_count=1 channel=[id: 0x17fe9fa3, 0.0.0.0/0.0.0.0:35786] timeout=300.0>) 2022-02-02 19:50:58,845 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.33:8091 during connect (<_realsocket at 0xccb548 type=client open_count=1 channel=[id: 0xf17a7615, 0.0.0.0/0.0.0.0:54745] timeout=300.0>) 2022-02-02 19:51:00,855 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.33:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xccb549 type=client open_count=1 channel=[id: 0x8205a1c2, 0.0.0.0/0.0.0.0:46608] timeout=300.0>) 2022-02-02 19:51:06,085 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xccb54a type=client open_count=1 channel=[id: 0x7ebe353c, 0.0.0.0/0.0.0.0:59219] timeout=300.0>) 2022-02-02 19:51:08,095 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xccb54b type=client open_count=1 channel=[id: 0x89831a9f, 0.0.0.0/0.0.0.0:58764] timeout=300.0>) 2022-02-02 19:51:10,105 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.36:8091 during connect (<_realsocket at 0xccb54c type=client open_count=1 channel=[id: 0x6a3a3fef, 0.0.0.0/0.0.0.0:33223] timeout=300.0>) 2022-02-02 19:51:12,117 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.105.36:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xccb54d type=client open_count=1 channel=[id: 0x1b6a4c3c, 0.0.0.0/0.0.0.0:34947] timeout=300.0>) 2022-02-02 19:51:17,309 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xccb54e type=client open_count=1 channel=[id: 0x54384f76, 0.0.0.0/0.0.0.0:35764] timeout=300.0>) 2022-02-02 19:51:19,319 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xccb54f type=client open_count=1 channel=[id: 0x230b8839, 0.0.0.0/0.0.0.0:60213] timeout=300.0>) 2022-02-02 19:51:21,329 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xccb550 type=client open_count=1 channel=[id: 0x1f791790, 0.0.0.0/0.0.0.0:39887] timeout=300.0>) 2022-02-02 19:51:23,341 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.177:8091 during connect (<_realsocket at 0xccb551 type=client open_count=1 channel=[id: 0x58346365, 0.0.0.0/0.0.0.0:36095] timeout=300.0>) 2022-02-02 19:51:25,352 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.106.177:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xccb552 type=client open_count=1 channel=[id: 0x0b27e188, 0.0.0.0/0.0.0.0:48991] timeout=300.0>) 2022-02-02 19:51:31,135 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xccb553 type=client open_count=1 channel=[id: 0x563c75a8, 0.0.0.0/0.0.0.0:60356] timeout=300.0>) 2022-02-02 19:51:33,148 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xccb554 type=client open_count=1 channel=[id: 0x2b7007bb, 0.0.0.0/0.0.0.0:35044] timeout=300.0>) 2022-02-02 19:51:35,158 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.208:8091 during connect (<_realsocket at 0xccb555 type=client open_count=1 channel=[id: 0xe2451cff, 0.0.0.0/0.0.0.0:49597] timeout=300.0>) 2022-02-02 19:51:37,168 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.208:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xccb556 type=client open_count=1 channel=[id: 0xdeb1d868, 0.0.0.0/0.0.0.0:55997] timeout=300.0>) 2022-02-02 19:51:42,474 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xccb557 type=client open_count=1 channel=[id: 0x224f9ab8, 0.0.0.0/0.0.0.0:52371] timeout=300.0>) 2022-02-02 19:51:44,486 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xccb558 type=client open_count=1 channel=[id: 0xd4ddd62b, 0.0.0.0/0.0.0.0:58245] timeout=300.0>) 2022-02-02 19:51:46,496 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.122.226:8091 during connect (<_realsocket at 0xccb559 type=client open_count=1 channel=[id: 0x8fbbf0cf, 0.0.0.0/0.0.0.0:38468] timeout=300.0>) 2022-02-02 19:51:48,509 | test | ERROR | MainThread | [rest_client:_http_request:838] Socket error while connecting to http://172.23.122.226:8091/nodes/self error [Errno 111] Connection refused 2022-02-02 19:52:03,428 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:735] RAM quota was defined less than 100 MB: 2022-02-02 19:52:15,930 | test | INFO | MainThread | [basetestcase:initialize_cluster:457] Cluster C1 initialized 2022-02-02 19:52:15,930 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:52:15,947 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:52:29,505 | test | INFO | pool-16-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 0.463310793889 | Cluster node | | 172.23.105.206 | None | | | <--- IN --- | | 172.23.105.33 | None | | | <--- IN --- | | 172.23.105.36 | None | | | <--- IN --- | | 172.23.106.177 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2022-02-02 19:52:29,525 | test | INFO | pool-16-thread-8 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 19:52:44,563 | test | INFO | pool-16-thread-8 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 19:52:44,576 | test | INFO | pool-16-thread-8 | [task:check:468] Rebalance completed with progress: 100% in 15.0719997883 sec 2022-02-02 19:52:44,713 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 0.350877192982 | 11.45 GiB | 10.64 GiB | 52.50 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 0.925925925926 | 11.45 GiB | 10.66 GiB | 24.00 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 0.788781770377 | 11.45 GiB | 10.62 GiB | 28.14 MiB / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 0.200300450676 | 11.45 GiB | 10.69 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 2.32004013042 | 11.45 GiB | 10.45 GiB | 0.0 Byte / 3.50 GiB | 0 / 0 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+-----------+----------------------+-------------------+-----------------------+ 2022-02-02 19:52:44,713 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:52:44,713 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:52:44,897 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1894] Creating required buckets from template 2022-02-02 19:52:45,381 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2490] Waiting for all collections to be created 2022-02-02 19:52:45,703 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 0 | 4.88 GiB | 89.17 MiB | 0.0 Byte | 100 | | default | couchbase | magma | 2 | none | 0 | 0 | 0.0 Byte | 0.0 Byte | 0.0 Byte | 100 | +---------+-----------+-----------------+----------+------------+-----+-------+-----------+-----------+-----------+-----+ 2022-02-02 19:52:45,703 | test | INFO | MainThread | [collections_base:collection_setup:164] Creating required SDK clients for client_pool 2022-02-02 19:52:56,292 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 19:52:56,301 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 19:56:46,500 | test | INFO | pool-16-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****..........................................**.................******......................................***...*****X | 28102.700 | | 26458.300 | .............................................................................................*X | 26458.300 | | 26131.100 | ...........................................................................................*****X | 26131.100 | | 17106.600 | ...........................................................*X | 17106.600 | | 14821.000 | ...................................................*****X | 14821.000 | | 13628.800 | ...............................................**X | 13628.800 | | 13137.100 | .............................................*****X | 13137.100 | | 10940.600 | .....................................*X | 10940.600 | | 10919.800 | .....................................******.......**...******...........**X | 16620.600 | | 16616.100 | ..........................................................*****...***....*****.....................***..****X | 25240.300 | | 22513.800 | ...............................................................................**X | 22513.800 | | 21540.300 | ...........................................................................****X | 21540.300 | | 18127.500 | ...............................................................**X | 18127.500 | | 17592.400 | .............................................................******....**.******.*X | 19194.300 | | 18732.000 | .................................................................*****X | 18732.000 | | 13842.300 | ................................................*X | 13842.300 | | 12936.500 | .............................................******..*******X | 13703.400 | | 11401.100 | .......................................*X | 11401.100 | | 10742.100 | .....................................******.**....******..........................**...*****X | 20290.800 | | 14573.200 | ..................................................*X | 14573.200 | | 12798.600 | ............................................******..............**.......*****X | 18648.700 | | 17293.100 | ............................................................*X | 17293.100 | | 17065.900 | ...........................................................******.....................*X | 22965.300 | +-----------+----------------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:56:47,546 | test | INFO | pool-16-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****.......................................................**..............*****X | 450.000 | | 288.800 | ...........................................**............******............................................**X | 648.500 | | 483.600 | .........................................................................*****X | 483.600 | | 120.900 | .................**...........******..................**X | 309.700 | | 261.200 | .......................................****X | 261.200 | | 19.400 | .**X | 19.400 | | 0.000 | ********....******..............................................................**......*****X | 470.900 | | 67.900 | .........*X | 67.900 | | 0.000 | ******.......................................................**..............*****X | 450.000 | | 127.600 | ..................**X | 127.600 | | 68.800 | .........*****....................................................................***.............*****X | 590.900 | | 147.600 | .....................*X | 147.600 | | 59.700 | ........******.........................**....*****X | 248.100 | | 167.500 | ........................*X | 167.500 | | 132.500 | ...................******......**.......*******X | 219.400 | | 190.300 | ............................******......***..****X | 240.300 | | 119.400 | .................*********............*******X | 200.000 | | 159.700 | .......................*X | 159.700 | | 130.600 | ...................*****X | 130.600 | | 90.300 | ............**..*****X | 100.000 | | 19.400 | .*X | 19.400 | | 0.000 | X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:56:48,437 | test | INFO | pool-16-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+----------+ | 716.400 | ...................................................**.............*********...*****X | 940.800 | | 383.600 | ..........................*X | 383.600 | | 159.200 | ..........******.............................***................*****X | 781.600 | | 702.000 | ..................................................*X | 702.000 | | 518.400 | ....................................*****X | 518.400 | | 40.800 | .**X | 40.800 | | 0.000 | **************...........**...*****X | 200.000 | | 40.800 | .********..................................................................................**...............*****X | 1359.200 | | 404.000 | ............................*X | 404.000 | | 200.000 | .............******............**......*****X | 440.800 | | 281.600 | ...................**X | 281.600 | | 208.000 | ..............*****.....................................................................***................*****X | 1371.600 | | 336.800 | .......................*X | 336.800 | | 79.600 | ....******......**...******.................**.....*****X | 500.000 | | 181.600 | ............*X | 181.600 | | 100.000 | ......******...........**......******..................**.*******X | 600.000 | | 559.200 | .......................................*****X | 559.200 | | 240.800 | ................********......*X | 320.400 | | 259.200 | .................*X | 259.200 | +---------+--------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 19:56:49,665 | test | INFO | pool-16-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 24321.000 | ...................................................................................................X | 24321.000 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 19:56:49,684 | test | INFO | pool-16-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------+---------+ | 259.200 | ...................................................................................................X | 259.200 | +---------+------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 19:56:49,813 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.36 | kv | 70.8517268242 | 11.45 GiB | 9.18 GiB | 52.50 MiB / 3.50 GiB | 834822 / 1678606 | 7.1.0-2211-enterprise | | 172.23.105.33 | kv | 76.3851044505 | 11.45 GiB | 9.33 GiB | 24.00 MiB / 3.50 GiB | 826882 / 1652994 | 7.1.0-2211-enterprise | | 172.23.105.206 | kv | 76.5428904127 | 11.45 GiB | 9.27 GiB | 28.14 MiB / 3.50 GiB | 827662 / 1652727 | 7.1.0-2211-enterprise | | 172.23.106.177 | kv | 71.0065359477 | 11.45 GiB | 9.25 GiB | 0.0 Byte / 3.50 GiB | 828587 / 1674123 | 7.1.0-2211-enterprise | | 172.23.105.164 | kv | 72.942400208 | 11.45 GiB | 9.14 GiB | 0.0 Byte / 3.50 GiB | 829052 / 1636203 | 7.1.0-2211-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2022-02-02 19:56:55,855 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 19:57:10,023 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:57:10,023 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:57:20,871 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:57:20,871 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:57:31,786 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 19:57:31,786 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 19:57:50,151 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 100000 | 9.77 GiB | 309.47 MiB | 384.51 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 50000 | 4.88 GiB | 460.50 MiB | 537.46 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 4070000 | 2.50 GiB | 1.74 GiB | 10.20 GiB | 92.5506142506 | +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ 2022-02-02 19:57:50,151 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2022-02-02 19:57:54,516 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:996] Doing collection data load during graceful_failover_rebalance_out 2022-02-02 19:57:54,516 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:360] Starting rebalance operation of type : graceful_failover_rebalance_out 2022-02-02 19:57:54,516 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:576] failing over nodes [ip:172.23.106.177 port:8091 ssh_username:root] 2022-02-02 19:58:23,839 | test | INFO | pool-16-thread-29 | [rest_client:monitorRebalance:1599] Rebalance done. Taken 29.1919999123 seconds to complete 2022-02-02 19:58:23,842 | test | INFO | pool-16-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2022-02-02 20:03:28,845 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:329] Starting a sync data load after failover 2022-02-02 20:03:28,855 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 20:03:28,861 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 20:05:33,209 | test | INFO | pool-16-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...******....................................................................**.....................******........*X | 40116.200 | | 39665.600 | .................................................................................................*****X | 39665.600 | | 30312.800 | ..........................................................................*X | 30312.800 | | 27327.700 | ...................................................................*****X | 27327.700 | | 24749.600 | ............................................................**X | 24749.600 | | 24332.600 | ...........................................................****X | 24332.600 | | 21938.600 | .....................................................**X | 21938.600 | | 20957.500 | ...................................................*****X | 20957.500 | | 18572.000 | .............................................*X | 18572.000 | | 18146.200 | ............................................******.**.*****X | 19190.200 | | 18698.100 | .............................................*X | 18698.100 | | 18212.000 | ............................................********.******.*X | 19158.700 | | 19113.000 | ..............................................*****X | 19113.000 | | 18002.400 | ...........................................**X | 18002.400 | | 17645.400 | ..........................................****X | 17645.400 | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 20:05:33,796 | test | INFO | pool-16-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.........******..................................................................**.........................*****X | 590.000 | | 572.600 | ................................................................................................*X | 572.600 | | 531.200 | .........................................................................................*****X | 531.200 | | 447.700 | ..........................................................................*X | 447.700 | | 396.100 | ..................................................................*****X | 396.100 | | 331.900 | .......................................................*X | 331.900 | | 299.700 | .................................................*****X | 299.700 | | 287.500 | ...............................................*X | 287.500 | | 273.800 | .............................................*****X | 273.800 | | 263.900 | ...........................................***.****X | 271.400 | | 209.400 | ..................................**X | 209.400 | | 193.300 | ...............................*****.*.*X | 202.000 | | 195.900 | ................................*****X | 195.900 | | 166.400 | ...........................**..******.........*******X | 233.400 | | 119.300 | ...................*X | 119.300 | | 90.100 | ..............*****X | 90.100 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 20:05:34,846 | test | INFO | pool-16-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *......******..............................................................................**................*****X | 1068.800 | | 1033.700 | ...............................................................................................*X | 1033.700 | | 997.500 | ............................................................................................X | 997.500 | | 997.500 | ............................................................................................X | 997.500 | | 997.500 | ............................................................................................***X | 997.500 | | 511.900 | ..............................................*X | 511.900 | | 449.400 | .........................................******......................**....*****X | 729.500 | | 708.500 | .................................................................**X | 708.500 | | 655.400 | ............................................................****X | 655.400 | | 407.300 | .....................................***.*****.*...*X | 467.500 | | 426.200 | ......................................******....**...*****X | 497.800 | | 423.700 | ......................................*X | 423.700 | | 406.900 | .....................................******...**.*****X | 454.900 | | 359.300 | ................................*X | 359.300 | | 345.800 | ...............................*****X | 345.800 | | 340.000 | ..............................X | 340.000 | +----------+---------------------------------------------------------------------------------------------------------------------+----------+ 2022-02-02 20:05:36,108 | test | INFO | pool-16-thread-25 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 16519.900 | ...................................................................................................X | 16519.900 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 20:05:36,119 | test | INFO | pool-16-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------+---------+ | 100.400 | ...................................................................................................X | 100.400 | +---------+------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 20:05:36,137 | test | INFO | pool-16-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------+---------+ | 340.000 | ...................................................................................................X | 340.000 | +---------+------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 20:05:36,243 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 20:05:36,263 | test | WARNING | MainThread | [rest_client:get_nodes:1885] 172.23.106.177 - Node not part of cluster inactiveFailed 2022-02-02 20:05:43,921 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4953] Performing scope/collection specific operations 2022-02-02 20:05:43,927 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5043] Done Performing scope/collection specific operations 2022-02-02 20:05:44,023 | test | INFO | pool-16-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.36 | kv | 7.1.0-2211-enterprise | 28.3838898802 | Cluster node | | 172.23.105.33 | kv | 7.1.0-2211-enterprise | 45.3678296093 | Cluster node | | 172.23.105.206 | kv | 7.1.0-2211-enterprise | 60.529732537 | Cluster node | | 172.23.106.177 | kv | 7.1.0-2211-enterprise | 0.61357375407 | --- OUT ---> | | 172.23.105.164 | kv | 7.1.0-2211-enterprise | 47.1639066371 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2022-02-02 20:05:44,048 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 0.0 2022-02-02 20:05:54,079 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 34.53 2022-02-02 20:05:56,211 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_Kuen2rZR5W7kyWgbGAXqWc90af3gl8t6qg8v3hUNDPeC8JwWTJRjEWwrXVEHFwMklk3TDJrrBVknNQ6hD2ALh1Ve%pRsdsAplzFtGpSe4Xsdy2EGmzrEYallyao59dR_pnR8ix42qQel6IaaPFBKYViNH4XopydKu5sqycsl4e-44-919000_RtqtLkKktLZrhgHRikNfkhpc3YOUa-YH0vB5M71%nn%G3sQWkw%UnU-JvUyFX-44-920000_ttl=0_update_MAJORITY_196_272_1643861149.19 2022-02-02 20:05:58,650 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_Y5zBLH-NPOvqdJ33W%PXPgHFnAfaqleTmVCagTXzyqjiAH7SaUlrLp4asxkMaK1rBu2VCYwsSgMI4dIUkIwWuLn7_5uq6EaFxZAUqvUQ9ma76JzFMFT5uaFXoGgW2OA5yqU6P6LplsS8kE_4cfVwftLrIhV-44-917000_a03FgUpmhKFJ768ZBOWg9NuGA2R60owere3IarxZ-wHQ5DTOEj%c5WyPf-jwI-LOAYxN92Gc45m0SNa6AWEpRNcq1P5v%%Qo9Opqb-n4hcMtyIeoxjskpe3EMzPAtsc-uXAOJDMN6jXzi6WCKEe4FumtjqXGwWxsWZsyfGy9T4Tq7tlTuoB_JxLVKPzfDAs3CS-44-918000_ttl=0_update_MAJORITY_196_272_1643861149.23 2022-02-02 20:05:58,650 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_6VxukfnuEyYOcefQd-44-924000_22Bi2ICqFwez%%pwRkzX8itpKQh8YEXNpjOiNoADMsE2nXX7zwfL7F4GIpkRAnXyBLKva1g-3rwOPxsMiExpCRc4bYL7VZoS8-de0rPxvapa3kQxBD6U4QofDgbgumPpkHxjeY54DiODY6Mm_QsiOb5TEaJ5zYQfiJmX4ggs2PckCrEuG%pZmtw66TbR1id%vXlQu95XtTF-44-925000_ttl=0_update_MAJORITY_196_272_1643861149.23 2022-02-02 20:06:02,630 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_H4bY%zcRx2Uh8c9LCnIM57bByeBMDdeNqb1GmmjyQGYboflilBl-wExc_8UDhy-zgOHRwASW8EJuHsv6F-44-921000_Dmu%BXURd1xEs_HF-gbN37vqbFUUYVKeA4WW0jSkdGKMYBzE3lfXZPpez0Bij8nU2ZMJ90ep4QwLlGHp3SxA-44-922000_ttl=0_update_MAJORITY_196_272_1643861149.28 2022-02-02 20:06:02,872 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_Y5zBLH-NPOvqdJ33W%PXPgHFnAfaqleTmVCagTXzyqjiAH7SaUlrLp4asxkMaK1rBu2VCYwsSgMI4dIUkIwWuLn7_5uq6EaFxZAUqvUQ9ma76JzFMFT5uaFXoGgW2OA5yqU6P6LplsS8kE_4cfVwftLrIhV-44-917000_ryyMCczR4EnhEiJzOUa4JwtQC2kysL-a_oeILH%xWeTCLmvYshYNAgT3EnDE9CtI%ah9TOOq1RyA0tEch_KvAslXOMrJmumTOrLU3D-44-918000_ttl=0_update_MAJORITY_196_272_1643861149.29 2022-02-02 20:06:04,130 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 48.41 2022-02-02 20:06:09,391 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 196 to 272 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_6VxukfnuEyYOcefQd-44-924000_6HUDW9pPCOmaRr0u2%xojYclwfRHpxkm4ghR7B2s4pBF83t7NQAs3iZZygj6Lh9cXL7h1ZCkjEuAZ3j32w7ft1IIxdsFVA%4tczns7J6tVFUtlfLnZti_rROMu80wYge-jPJUYKgWC2LI-44-926000_ttl=0_update_MAJORITY_196_272_1643861149.43 2022-02-02 20:06:14,167 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 58.6 2022-02-02 20:06:17,167 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 100 to 196 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_7CIkKzuICjfOjALc9hhp86johmmwH4BKGpMYHHIQcAPHwqypbFoVXyfudRI7skf-uWhvVSVHhqJNlxiLWMzWju-44-926000_x5tJRI28yqI%R9N0EB_x3Nk5sAEwMhSlkn-2ilkjWdI2S9GiXX-NCVD99PqDR_nwZLjd77gpSn6rSCJP8MIRydu8xKszw2Y8HKW6m7UR85J1YZOe1RWaHU9Y-Vjy-ME9iuqt-D1TQCTEnpVENhjHROYNO%jbxbD5S2ED-44-927000_ttl=0_delete_MAJORITY_100_196_1643861149.5 2022-02-02 20:06:24,213 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 62.71 2022-02-02 20:06:33,321 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 580 to 656 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_bucket2_7CIkKzuICjfOjALc9hhp86johmmwH4BKGpMYHHIQcAPHwqypbFoVXyfudRI7skf-uWhvVSVHhqJNlxiLWMzWju-44-926000_QYQgh2xyckReGYWMCaBO4G2BzCEvkwe8B5cxrla4ZY7zT_4OAzlzR6_4JKDBPvLxYttr5yrWu0KSdcPs5zzRwn99yg%1K%pnwt_Zg16P_xSxS7R5UwdXEKKZCcjp1ts0ggXWUFgYH9pJWgi6jVUiowRwlGu-44-926000_ttl=0_create_MAJORITY_580_656_1643861149.62 2022-02-02 20:06:34,259 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 65.31 2022-02-02 20:06:42,509 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_QYnP8VF8SpL-QFSgGcjfpg_nSFiHH8utmJtpFrgiduqjAnigUD%3Lz0xLLnNpcAdiwmtj3FMKSST8WrO6zxZAnDsbh-QicTH-oiggUZ4YH6xvDw_kAwpW8qrqbJTFjOBqQgTY3L0ip-W0KAiG11PniYGSB4iuvc%4yNOf67WayImBeavEYB-44-960000_ttl=0_update_MAJORITY_6381_8881_1643861149.68 2022-02-02 20:06:43,440 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_p8tx1gRKsyJWa1c%pU7pN0pnB9Yb3mqrqYABQpRii1PEKnSkdTuK3cCq9fO5Runhzbpkurc4BW4awcb1N4y0RBl%C28_eMR02PM%aV0ouRwGeUd5h7MWeZjcl0EynAX9dKNe8wKn_cmEQ2FOG8r56ILR5p9mggZrUo209pJ5FmLITl40IfXYz7Lr%nAzB2-44-945000_ttl=0_update_MAJORITY_6381_8881_1643861149.68 2022-02-02 20:06:43,595 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_UNoXXKkBAwYB2tdYRjiOLetynMY-_NAvSPJK3-8CRE3Acvpi6n6TxTEWazOIJzf9Ana6lGbJAWD-qlAEs25JmFgdmpJabI0nbnCWx-ymM2wtx-44-949000_ttl=0_update_MAJORITY_6381_8881_1643861149.69 2022-02-02 20:06:43,595 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_RCXABN6tfDz38PGbt1wLxwwHZFa-mXH_5ujbDKqIyhRfMKPw8astGl8ZfAZQewKiOFavVLVNZDEMWijRpqEQCDs9%BgXApnbPxvuIi5sVbnIsDHx964tO3v1xa4cKGTnuW9-44-969000_ttl=0_create_MAJORITY_18884_21384_1643861149.69 2022-02-02 20:06:44,302 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 66.99 2022-02-02 20:06:46,076 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_QvBwml4OGeR25YPZ%0p1rKbXRD8nZdlsuv1HyD7XJpvIR6gm1x5sQrGwHkJkOLrfe1xmX9te1iCeX3ewcROqB6SE6X3hfFxr7JbvdfLxTllL4_r0k5FJWPx8RI%NJpj_I9xYsXp582q550D8xJxVSkJz3Ibm%NZ7YNT-44-965000_ttl=0_create_MAJORITY_18884_21384_1643861149.7 2022-02-02 20:06:47,109 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_WnRtxHsR0gvVXOze%VFz-88wUhxNRUZfkg34a8q1yIeSLYrCgt%ExQ7yUSRbs_WTO3Xi8ISKmoNVxewu3_6BtBeRoluiWVqNczINp7DNHg2Sd3GZz%7QhqO8LYZkBbmoC4XMnc0L_pgHo9NLvpIc40dJ0xc%a32sV9weJtotlMj6Gkw6OJaeXJkjGDBZ637WQrHf2ZMv9fufcoJIKu-44-961000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:47,776 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_iLI9WgnuJz41vA-44-943000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:48,470 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_u1hEGnaLBhVi-7hp1trUmGj0h59oPsY9%ChkTpol-44-969000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:49,023 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_NSkct%HI4oait-HTg2k787QFL9bUtech0GbwKnQxo00y179d7KA_x31rYo-sxscLW4Y7n56QM_-JXLrXVSSYpi-l22wiZRpKxEgyKwQRkWr3iT%eWKzpJNSTjidol70Deyr4u%cla93DoMMFKuDK0DLS94fjI67NKxln9u3Y-fDr46XTauGIIFT_ttG1XIbA9NItnI8T_ENYqK-44-954000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:49,023 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-44-955000_uiR5h622CnKndWWkKlS4ZYN7AJT%8KXpTHGfCpKWfYm1UWKBMAn5PmwJVaZ4PG5BStk%C-Tav-BLuC1TeO_t-TZUBFScpdXhqdSef0Zh1z5fr0XzbG4857vJXUcC3HmY%yOxE0uPbAvLds7wu0K9QJL3A9LKoXG3ie9gR74DgWlTXbe--2EN%3WDDD5otSymk0vNcyUmZ9sJ_Pn_3cezqv-44-958000_ttl=0_create_MAJORITY_18884_21384_1643861149.71 2022-02-02 20:06:49,023 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_K7b6H1E60992AHVqC28IxXJw3_5ju-44-947000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:49,023 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_vau-CndB8BNbKEOrErJcS4fgGL_X_CIafo85WH7EH-OfNVqX2zuxbqJYQD-uA68MYvLQnT-1ZjzzhbxCO-44-966000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:49,117 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-44-955000_EIT_r2gUh4Z4clfk-mXopG3%SvxbyKU7IaqE5nQjGB92V48kJ3w1Sa0Ymuo2XwphWJtmqz-44-958000_ttl=0_delete_MAJORITY_3256_6381_1643861149.71 2022-02-02 20:06:49,424 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_d4BaMPwHWvIwCDg%A9v4RbgevIMAQgCthFgwZsUfhgD5QFInCWBteeLdPbBVCzY4-ELqArL6p65_YJrxamjy-Y2ZpbUGYThZMlwk2bYTwlGXH-44-965000_ttl=0_update_MAJORITY_6381_8881_1643861149.71 2022-02-02 20:06:49,483 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_1sGTQkZ0ttNfgPuadfXxmYgQXGDMo-pjs85OC7oGL-44-948000_ttl=0_update_MAJORITY_6381_8881_1643861149.71 2022-02-02 20:06:50,253 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_7goEqujN9KGhugYVPC3Pymo5bZhE-IlLebOsQqsD2SgQlWruHxHD7lHYOX4tPC-5lm6LPharHnIpmiBrr_ETY9-FKMmTqRmQq2vNjqKDow-Y4YEiNY3WoF-TVE2TV__q9iShEXOM3i0OTtIlGly0p0vsCCqbzR1Q9zwTeJAcGfKRk5LPUJ-44-949000_ttl=0_create_MAJORITY_18884_21384_1643861149.72 2022-02-02 20:06:50,565 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_eL6jfEd%f6HD41i9_L%R0mvgcIzW8qQCP__g_fTpyppzJyAuVgAG_d3gDEpXR2xW4L3MiE0K-%mX0rOhi7favV30yWl9c-FgmXnqPwRw_ysQBCS_eRojIS3NjKiTgTEBGtKbhAVDHdBvRRd_-B8uUQFlJVCIxFYQDbs-44-942000_ttl=0_update_MAJORITY_6381_8881_1643861149.72 2022-02-02 20:06:51,792 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_tB7Ro%gyWPuzVpryqY5sH6SlcgLLq49LcHG_kDRafiZ4ZT9WmptvHLuPXHtHkXtJDAP54xFH%-qNTULKmfYGkL1tjlJ1UUkebyNEokydkQoTMJGRBfgmffvsVHsftAE8QtCBPgBIg4fgMZX_Oi2mE-p5ch6rwjBefwckeOm9E8rm7IrnSkK1hHI6iiDxBgTLJ9Fbom-44-954000_ttl=0_update_MAJORITY_6381_8881_1643861149.72 2022-02-02 20:06:51,793 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_SOG%U6-44-955000_ttl=0_create_MAJORITY_18884_21384_1643861149.73 2022-02-02 20:06:52,970 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_7JecXTIYnot%MgfXyXTmAlBPCiJelzdJfJKcWjJy5U4mczQ93hqnX7-AVR-4FDIyGRcbsZ4UWWq%CS-fre3NT-_T1zDLfS6YYzhJGB%XxALvvSFmY2As5ie72sVsMsS8RQ5s7Ua-gUbsOPrf4CR99qN8bYPOKyOBVn0fMuUU0rZE9Znkziv-44-969000_ttl=0_create_MAJORITY_18884_21384_1643861149.73 2022-02-02 20:06:52,971 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_2kfUhnhqbD8NNPYh6Gva7rqqjki%7m76ECHLQdu8K3lqX9Y5-aw8-44-949000_ttl=0_delete_MAJORITY_3256_6381_1643861149.73 2022-02-02 20:06:52,971 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_ZFe18IkMKsgH9QZOPZC1W1PKeICSizfISvkFzJl34PRTs_xnj3g%atLME9c4MjvTwfEsEhHJXwJCYyUK7G6NpRDF-CmojH1LJ%gh%IdszZgWXSkoR7K2j95bc-a0XatLQp34wM2RNYFQWT4-gZ-SV%S%oZHxxZ4lFRvZe0s1gN-44-947000_ttl=0_update_MAJORITY_6381_8881_1643861149.73 2022-02-02 20:06:52,973 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_sY-UhJ55Tmz%i3wbWg7Bi1GJ-QCyrtOXX-Wzo33j6nLqmjDzc7MQRRNgbAzsM_Et3RyHrHcf_P_Guc9i6D2cx3l5RF0MyGGLqKPC73bAhRhKvwfPW24dscGtJq5cGV9ZAOZnftPQjBr2N9gtyKfIUewtBFZps3TP05eu9-ElYE7G88Z-xGRumPIz49udHXh4LNvG8VNrsFL_S-44-951000_ttl=0_create_MAJORITY_18884_21384_1643861149.73 2022-02-02 20:06:54,167 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_gzfeUopt_YwV-b5SXXBGOoJNh4ggVdWjBOZ-44-962000_ttl=0_create_MAJORITY_18884_21384_1643861149.73 2022-02-02 20:06:54,346 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 68.84 2022-02-02 20:06:54,938 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_ILdcaDzh6ddDdF5Nfo4eO7XTM-5SPx29UuvuiSel8q3fFnLFjjSBpllqH4lxdSwf9FdlCk8iCzGibudGccQT3eYE08QNQN20ri%VobxCw2ODE6gmyDUqGXfz%GuC9KBLafNrJlQJB0Vqz-cLqG82H3DzU03h14gxo-bqtk8d8a9JokI0-%xMSy9zOXL1RFTJTWZvLvahKId2ZlQqdc-44-969000_ttl=0_delete_MAJORITY_3256_6381_1643861149.74 2022-02-02 20:06:54,940 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_PZ%JsKPT%c%Bl2Hb0FXcRrgpu8SvWrzOx-eQ90JhHxLBLbbdsxHMBQE32Nto84ykUEbScO_5AdHAnaAp5KR7djIZrqP18xYizYAao5VJ8bS1nvY3okhL1QFBczKnFp96cKCikIjgDQT-44-945000_ttl=0_delete_MAJORITY_3256_6381_1643861149.74 2022-02-02 20:06:55,105 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_AZqGFg-hy2aiASXYMbasJ422d1uwm7RdjJGSwiMF-44-959000_ttl=0_create_MAJORITY_18884_21384_1643861149.74 2022-02-02 20:06:55,107 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_YKlyyIFBSN1orCHUZ_3Sm8sxqkb2h0d6kfoIa4MsTkHNc93GTok1OUvooESWcGjqRt2JfwbDq9oT690yL8M-44-950000_ttl=0_update_MAJORITY_6381_8881_1643861149.74 2022-02-02 20:06:56,223 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_aTJNrFf14omHxgE_y4q1zdjl_nUBb8krs2M6b9IBG1SGcfIdwZBolUYs3lQ1I6Ko2hO6PTqDivu5KbGoNS3sV42Bet-44-963000_ttl=0_create_MAJORITY_18884_21384_1643861149.74 2022-02-02 20:06:56,482 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_08Kq1K%%WhoVLUl1dqIWpUORWGCoE0zMdPoBzqdpYbBwZWpdWBOyZtFnavT4GwsmCZEAD6_dNOusfq3KxNXxGO19AoUgewtaVX%V2_bOT_zTUiJ_pQB58_7hw4qjmSlT%xMDJCYZ404s7HQp3WGxWP6ibvpZw5-44-961000_ttl=0_create_MAJORITY_18884_21384_1643861149.74 2022-02-02 20:06:56,482 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_thb_4JiGBBIBs%XunLR83_AovgZpLjRpXI3WzLInY8Cj--wm332X9rW1EvdNMOARwiEyg9LUWKSAFO4XYpTXPN8kY4yWTOQdoagDtV-44-952000_ttl=0_update_MAJORITY_6381_8881_1643861149.74 2022-02-02 20:06:58,384 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_e5hmcOuluotbnmiXIUg5U%hbwg3DpkQYWZtTym41NRrnZ5fy2TvBKJ52EMo37Zn6U7_a_sQk9MSQ8IsTiMWWk9Cq%pUa0osekJci4ctwiu17sM2pAe3b-FqrDdM-l-MjGZoSsIVk49UHxllvLFv62wtQOEy97dFB33aicnN7xojRfiS2lomYGn-44-941000_ttl=0_delete_MAJORITY_3256_6381_1643861149.74 2022-02-02 20:06:58,385 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_49qOJfOkdDkouWflxH8ntPUe0VbLV-otnljbaIpGFXL7wNB8wrbBjzxj8%CRlnxjp-1u%rVnsX60%v-tjzmsHhwCMRXYyLaO6_Vy9aNm7GFvY-KTRC6npMbO36k3U4dMngdJtx%0MRQMdFibdw_wsluhAGTxBE2jk1EoAC%EPReiOd-uNOEw9Ov2eeXju_GHQro02JSN%as3LklX6Q7yr2xpAJ0CZmykZW-44-954000_ttl=0_delete_MAJORITY_3256_6381_1643861149.75 2022-02-02 20:06:59,522 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_VpuUXBuJ41Xdwrtmdvx58ZUaR-44-946000_ttl=0_delete_MAJORITY_3256_6381_1643861149.75 2022-02-02 20:06:59,589 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_rP8E-44-942000_ttl=0_create_MAJORITY_18884_21384_1643861149.75 2022-02-02 20:06:59,591 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_neh-rON6vkUno04lEn7zXbzCN5yqUWN%OrCnUqOdrhObPV6Mzoq9ZraCwAWCDrayk-dZpIjCA_-UrkSPEsUIq8pYV_BD1T2i%WfjVrSt9S2MaGqp8_kfZSjzQLqBQPaLa6%lVPyZChyuUNdtXmXBW4SMaUMUQDD6B1_kf5hZSogTFSYM-44-941000_ttl=0_create_MAJORITY_18884_21384_1643861149.75 2022-02-02 20:06:59,591 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_Le4L6DSo0L4Kidap8HsClfoCUEZY1HIHfdASXHZwnYbaODxNrgXEfghXmehBn_MwwypWogZ8_F3iR-qC1kZIEjBOt5c8oFHTdHQthdnQXDqMb%9X1pNlRx1dWfX_akgb69Unpl9VQmuO-cadg-44-947000_ttl=0_update_MAJORITY_6381_8881_1643861149.75 2022-02-02 20:06:59,591 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_g0YmCgrxY5H2Dm0qeHAvUqsFa_My4K2PA6e7Cm%gyJr9p8Eukbg02XpJPxblIVZ7TkL6wqC4vRf36NTmnkDfn7JsuNQeUCitHb30e2NtvmIhOp2UvxixMZiN2xXChkCF_iqTfERY5JQtji3nvoT-v1BPuz9DuX%lNwYNiOdB35oJa_7E6TGcsgAZXXHQkPsi-44-941000_ttl=0_update_MAJORITY_6381_8881_1643861149.75 2022-02-02 20:06:59,592 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_RX57TAtOUVaVpEMtg_kVnjWqAP68pwG94tkDDG_18WWMj14OxBxsDHDzjLSpuZUM8p4dKDQLBPV01yHoKXG_h6GSdfWVMqoxDY_bwrbN-44-961000_ttl=0_delete_MAJORITY_3256_6381_1643861149.76 2022-02-02 20:07:01,509 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_w-KwKQBS0r9Sq9WVN_UnAppoDbighi_847pm7diTUUtUs9xqYsa1fMqQsHlKyXkftqbygSTqlsFm3R5NXL3Sx6hd6X%AIWNcBsJWA9P7T_rbbT6qBXp9nG4xpfyH6h2kwVd-44-954000_ttl=0_create_MAJORITY_18884_21384_1643861149.76 2022-02-02 20:07:04,006 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_Js-Pp_PJHet1kgSKlPQESgkuzjSESnMvJV7YUHGFB4wYBdcu8Xlj36_sj3w5zWPM%0jIOdC7I0bZyf9cU4vgKwNz3FQQYs3aw78k3gXqI2PsCl1AtqFBNJC-wYSAkefF20SrXWCG3gMex3QcjXmEDe3b_6_Tk2zx3cIYjIBsZvKCXa3Kdp6am61fY2PgrWjO3AnenFyb4J2yU-44-963000_ttl=0_delete_MAJORITY_3256_6381_1643861149.76 2022-02-02 20:07:04,006 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_1HHyNUaDd0Urs-0iVdFN5Em%mscKEchHz2KAaLa2qdMgO_MIxK5FMUhAtr2H7-yLvE7Kk2RqyH-QGhthw-44-969000_ttl=0_create_MAJORITY_18884_21384_1643861149.77 2022-02-02 20:07:04,391 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 70.41 2022-02-02 20:07:05,683 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_ucIIj-44-946000_ttl=0_create_MAJORITY_18884_21384_1643861149.78 2022-02-02 20:07:05,684 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_CDROhTBLNzWMXLUdbAJX1VQJjPJ8AuE6zPqPSIEO0SpeAl83y7uu1qMJ64mhxri9uRkJWvpBkYUJLyQcsfxwDucTaPZ5gFyIDPmdxpQ2ZjJmTgZcCPbvTdgdMT13V6SyMgj0WNVQbomb_oDDIQrJKTjORlsyn1UGE2LkIZJTyvHQ-44-943000_ttl=0_update_MAJORITY_6381_8881_1643861149.78 2022-02-02 20:07:05,684 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_xhocbUZmakPwC%Y7qRPa1IXwP_77ijFgUxUNEIMgyhjtLaTXqpOYS-44-947000_ttl=0_update_MAJORITY_6381_8881_1643861149.79 2022-02-02 20:07:06,874 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_OPvbsnlRy5t9_SKCWsdL388qYpCMEn9C6cCQ0B7jjNhJROHHGghlmCXxjeI4DS_Qr880%G1o_cn%RejMvzr_TxQ715TchY7vAXo2oCAj-DoDoOl9Nf7LxLPlOu0iWJGfJsM-44-951000_ttl=0_delete_MAJORITY_3256_6381_1643861149.79 2022-02-02 20:07:06,875 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_IJodTcMctQxMvnZP2oSR0q-p1O7lucMJJQNcDDp_NgqSOuaC8axSBC6-44-942000_ttl=0_delete_MAJORITY_3256_6381_1643861149.79 2022-02-02 20:07:06,875 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_HSrq5vwDj9-K6PFasqp99yMfi-44-970000_ttl=0_delete_MAJORITY_3256_6381_1643861149.79 2022-02-02 20:07:08,615 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_ClUUnu6lkK1AvFvwhGTMjw_LwQLgzqo5MmComdzkDsswtSikh09p0DoGaTsUM5wkvlNbTiVnRiG_4MCEV%BDKblWj1%CzxeM8jn_VgXP732a5Z9moJhAqr5LTNL8ewRs6keo_uFb7gRytxUnfXJs9_-KW6V5dIuSihvm_ce4o34DzKyLDGGmvUcw9CJLyCy0s6z-44-970000_ttl=0_create_MAJORITY_18884_21384_1643861149.79 2022-02-02 20:07:08,828 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_Q%2pSJFQn9DQDLEj0ceZAShtZNzcv745QjW2PW8ma5ZSHHMUuDs%P%CWbOKEbV0sTZRwhQT%AVlj9GkZDVbdfF5Tfg%u8BzUcMNXeRo6YJ5Ysa0mZALciubpLhx-iz0I%Nf8uO_P5pW6pgplZ9hnaAB4AC4Y9xY7H73_OsqCSY-IYEl-44-963000_ttl=0_delete_MAJORITY_3256_6381_1643861149.79 2022-02-02 20:07:10,286 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_vnRtZxZTkDNUvwCIsif272wR2tIXN4B65Rd5KYt3gloDnC4UbDpXX7Zfwsm168K76U6_r3b9qYPwavD7iUTIEPNYkvGO1nuzIV_stlMEKPehl9pdCh%9WSIxvkbb8Sz-93Riv6i%dSapS_T7HvyqnIcibqINp85-CH_pkspGB7M-BHsS2u_rsoFZjSC5O8fbIypjQPxx-44-970000_ttl=0_create_MAJORITY_18884_21384_1643861149.8 2022-02-02 20:07:10,286 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_o774Fri_Hht2bzsNlJdSB41sB8MLENhc_OtudMsmrOsP5fAAFVcAiFpcqPV6Kx6Aa2JPV_nEEDWH_uOMrCYmFyd4h2M-9zZ%2%WT_%GwjTfdOO%Ilor%b%cU4ehKrU658dNCHeVi-Rvr8ky889nnZ%BUXVpS%z_SDgu4JH73EtBQwVS%jq6cKLEC7Xti7UIIo5wBXUrVE_0Qya%VzDx1iSww6FX9uQx07Z0J%HpT_EirXgkZJ-44-945000_ttl=0_update_MAJORITY_6381_8881_1643861149.8 2022-02-02 20:07:11,869 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_Dy7kVJ-44-946000_ttl=0_create_MAJORITY_18884_21384_1643861149.81 2022-02-02 20:07:13,354 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 4 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_GAGFEM1Thvzc5bahxz2Mh1uyKiqTpQ_B5qEhsQPbSbuoJHXoAEVc_RziGGKLtSBUOiY_DbXFifTScaJCYPW6uKxd8NOL16xZ3P4aIJCe1lz3jchWjbq1lAORxl4XOtALasLCyd-UggXOgHaY9UJiRhWeXyB3UEL90jrcX2ggD_%vmW1-HGoqS-3V8hwvjXRgOYy2proI4RF5-44-960000_ttl=0_create_MAJORITY_18884_21384_1643861149.81 2022-02-02 20:07:13,355 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_gxxxR2Px8gfVz1IMf1r3oHvrsupRh0qt3p5J2juPEZu_j9%Foz3zCHPuYWb7NJ7sysQQe2Zmgej7GL-oNAN2Q0tfq7NSgKPivZyAgTbZVzZTW21rLH3afS-44-960000_ttl=0_create_MAJORITY_18884_21384_1643861149.81 2022-02-02 20:07:13,355 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 6381 to 8881 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_NeonB1qCdonSFfrV-sGU4xK2VCVN%6XjUx2ZU50h9cdKwmpvLTl4gnluuWIIRZyVrke4a42YVH6Hj-RkKZHaegyZAyKzNfGfe-U0efKPDXjkbHOHejM1FkZG-0-yxhSc0NvT5kQV4e87F4hI4vI_tUBeUl42rXi4cSD_y0Lh354XYkfCt831Njl1Al5jZw2Or8_ss6xrmU-44-967000_ttl=0_update_MAJORITY_6381_8881_1643861149.81 2022-02-02 20:07:14,085 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 4 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_aOi346RI7BGSZ4Y7n%HSw_XPB-66XVUD_4GypZ9AGRcLvJ44EIAL4VJtMkEXuPs3IaZ7i_rNrFs5EvgJJrj-8f8oLGCiXbCCfBfhv1Zea1lOTU-44-950000_ttl=0_delete_MAJORITY_3256_6381_1643861149.81 2022-02-02 20:07:14,086 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-44-955000_ZMlbC7PYTy1Sdn_8ufcVNImCXYwZMZx1YBD48b-AOnSQMffh6i7mPtg9TAVRKB1E01Qn6TijGuNruX6sUdb7BEQcRlwiJARJFZQY2aNteTYdzabxGvgGNKgZLL%YRdVRyPte7VCG1a3zGDgO4lyP2uzBG0AXw1naYuCORcT10tdDHVLFa%SQJJ6FB2PYlQD8LdtUbxvpI-44-958000_ttl=0_create_MAJORITY_18884_21384_1643861149.81 2022-02-02 20:07:14,441 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 71.82 2022-02-02 20:07:16,161 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_DbT0erxl7qcPKpPDI7D_Ts1L4hwxzuREXVJ9RGgLqVnEvabt4P1qYIUkGyO%s-NdmCvL%xDObZAOpH%Poq4dAEHalqHf2KvS6cjOWpATp9eQDOucO-9n9e5%yt1a4tuy%24-oZPZGLpjLMSAl%%AI0jziG9JRip_uMjxoIgsyiaJt039iFxdtFi_KPy14KveY9IkPZcaoIspGQnrdzyDbqs%SURLKWcsLa44d3iJb6V3w1FQ-44-950000_ttl=0_delete_MAJORITY_3256_6381_1643861149.82 2022-02-02 20:07:17,028 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_5Qb1555D-QNUCfFoXQUufqSio8FmUL%RQjh9F%aVaRV8092xdN4Zd%zds9hWJ-R32q5Odx_glvpNIl0H7nvZ5-X5urWIlxjJol-44-944000_ttl=0_delete_MAJORITY_3256_6381_1643861149.82 2022-02-02 20:07:17,769 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_DjqcgczWED3y%5XL7%bae%e7Ecu%alLkHlgqX9VSxfpYoE3sHzc8q-44-954000_ttl=0_delete_MAJORITY_3256_6381_1643861149.83 2022-02-02 20:07:20,956 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_YKlyyIFBSN1orCHUZ_3Sm8sxqkb2h0d6kfoIa4MsTkHNc93GTok1OUvooESWcGjqRt2JfwbDq9oT690yL8M-44-950000_ttl=0_delete_MAJORITY_3256_6381_1643861149.83 2022-02-02 20:07:20,956 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_iV5YrAce1FT-3EJwPtGz%sFNX3tz6_vn-44-949000_ttl=0_create_MAJORITY_18884_21384_1643861149.83 2022-02-02 20:07:20,957 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_NklaorkV-9Ze0Cg4hFedp1Q3t0CZOeJ1zcLOHk6MOnphHCYHr_-uZkFxxfp_Buiyaj%PJT%p7k8nRwLx8VyPVhLamfIgNN%eVfsjxh-F6i6no-3M8CYUvVr%sfxMyWsOb%%afG5r1IaI6Nzz1PWfC1XBow8BCv%SttNayNMyzUavjTR01UDy%5as8hNLe3oR05Crm9gDpf0cCpASZdJ84r1xA-44-944000_ttl=0_delete_MAJORITY_3256_6381_1643861149.83 2022-02-02 20:07:21,926 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_XMW-DIVlReptUpsZidy9YWQDvi1p-UNEDYAI1ltFH4Jpq7R-SpwaG3Aq-44-946000_ttl=0_delete_MAJORITY_3256_6381_1643861149.84 2022-02-02 20:07:22,849 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_68daRtUDI-4NvriskcInatKEm2ILnPxOvZwg2UsE7jLUpkrlpO4mMNndyA68SkbbKaoPbvYrtzhPJ1G2RHiNKJkvAbe-3QZ75Wq-44-968000_ttl=0_create_MAJORITY_18884_21384_1643861149.85 2022-02-02 20:07:23,858 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_5bU4ViIM27JuOlO_vg-44-965000_ttl=0_create_MAJORITY_18884_21384_1643861149.86 2022-02-02 20:07:24,479 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 73.34 2022-02-02 20:07:27,256 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_C6r_5umgeQrB2UblNlN4jIl8BQ5hmoi%WEwL_BtfGihk%uRhn4T1ocWT2YV7r67ZtoRPtHpVaLcr13hE4ZAu84vnrrgw_rNEm1zqtNDBKd0HgSb9ttOic6xVLXL8qnfJRVW_AOErbc0rgQIzE%qJy2Fm_sgbcHPT_kG42t2%Gzh_3dVXMT7UUsNhyvoqv-44-970000_ttl=0_delete_MAJORITY_3256_6381_1643861149.86 2022-02-02 20:07:27,256 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_wFNE6s3VEi0e1huY-2YzWRzCbQKki4xEzJm0-4yGuyhFRDWsRW8w%fjmcjB6Ugj5Bbz9mUvQLZMol-6cu302geDEbgfn%bz%QBs0-wlTCyYmSauiG64fIrZQ3Tmy_%-44-959000_ttl=0_delete_MAJORITY_3256_6381_1643861149.86 2022-02-02 20:07:27,755 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_UNoXXKkBAwYB2tdYRjiOLetynMY-_NAvSPJK3-8CRE3Acvpi6n6TxTEWazOIJzf9Ana6lGbJAWD-qlAEs25JmFgdmpJabI0nbnCWx-ymM2wtx-44-949000_ttl=0_delete_MAJORITY_3256_6381_1643861149.86 2022-02-02 20:07:27,755 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_c-44-963000_ttl=0_create_MAJORITY_18884_21384_1643861149.86 2022-02-02 20:07:27,755 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-44-955000_EIT_r2gUh4Z4clfk-mXopG3%SvxbyKU7IaqE5nQjGB92V48kJ3w1Sa0Ymuo2XwphWJtmqz-44-958000_ttl=0_create_MAJORITY_18884_21384_1643861149.87 2022-02-02 20:07:27,756 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_hln_-r%Hm2zWqBPbZqA5BYxwZomdD0bgRRD5y3yBZRtk2sHIuNRp4ov7LyUQ_VpVrZng59IpVyMaRx3tfwIVQb4fnCNWQ3fWS29VI%wUjyTaXZGKY0Pf3wqc38FdLgnwLCIlApI8Fq2a0%4A-44-942000_ttl=0_create_MAJORITY_18884_21384_1643861149.87 2022-02-02 20:07:27,756 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_Y-qgZgTNkAKd8mwLvSi-QY1sobDZhyGFg4SKnamoGySkobary%JOXCR7ZI8eLSQb2damrOAF%ijAOmBkXBKhBh8gL1zY1IG1gin1-44-954000_ttl=0_create_MAJORITY_18884_21384_1643861149.87 2022-02-02 20:07:27,756 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_E8RNzwspJi0ORDJ4ByP9RNOB_ikjeiwWVCblrMtEzrqJ4QdaGJERwr1nXqC6Xc5HF0fNS64%Z5BhYCClfg7UXt-lHnkPY5Y-Mjnce-44-952000_ttl=0_create_MAJORITY_18884_21384_1643861149.87 2022-02-02 20:07:28,845 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_NF7M4l2Epv4w5EpFfQyCG5SJ5B2J_JGg6u21fSxPTxBda0z53TApLgqDnxQTHHsEc%KPyoi8r5lUsxdLQ2J7Cjt1FGI-t1c2oFUb2iQ4qsPjjlEEJUmly0wj6J570eGIkAKEKfHnCtDxg1P6Eh71ShFtvelwxp9-weYpo7-v9qT92nDfLqDsviedj-qauSDxJ%zeXEX2qKEUGNbX7zWtZiN1EmfsImvrGMg9Ia_FF-44-951000_ttl=0_delete_MAJORITY_3256_6381_1643861149.87 2022-02-02 20:07:30,015 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_LngHxlvtZcCLt_2CljvUfkcQkO3sWt43rUYK%Q-g%MUOBm96l0_Za8PmQ4OfAptGF6e0vnIFLqYdrnqUQzkWRZPOoY6J%au6Vj831UzTJTbCoirHR_SW7paNuAYWWt8lSgd5-ZQ6S1JIiTWyrTQ_jQZIX5-1MClwuxRIahDbyCIioDZVXNxF7JSMXmo_oZruAAz3uV0YOlLfHrgHrQsA2m_58oVNHAbhgUENR1-44-961000_ttl=0_create_MAJORITY_18884_21384_1643861149.88 2022-02-02 20:07:30,375 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_BEks27wBcxdMnPYv4CLOtx62OU9ILtHLVxSCfZM75LV7dChvq0DZefIjuo5AYgiLAUW2KbpRr7tDCAuQmjlT6Xe6X-Jh_FtE-gocmFgVJB0AN3N1z11K4MH44-Ah_LYbm_khy%G_YQ%rC47B_skZF9STbRpdPAtzJC7J5r-VDbztphIpw_8%ESeSfCsDVXGDCKDzWPm2KS7mV0i7CMtoaPV8FM7Cfu5FuR-44-955000_ttl=0_create_MAJORITY_18884_21384_1643861149.88 2022-02-02 20:07:30,624 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_BEd-WumBHT91CFwrn5h1%3qE0hbSKo-WM4f8k%7zCZcm1wjqbQrfNDrbdLqRluu8RD%TaihkjeUjXfDLEDLLoZRLDC-44-962000_ttl=0_delete_MAJORITY_3256_6381_1643861149.88 2022-02-02 20:07:31,528 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_Q%2pSJFQn9DQDLEj0ceZAShtZNzcv745QjW2PW8ma5ZSHHMUuDs%P%CWbOKEbV0sTZRwhQT%AVlj9GkZDVbdfF5Tfg%u8BzUcMNXeRo6YJ5Ysa0mZALciubpLhx-iz0I%Nf8uO_P5pW6pgplZ9hnaAB4AC4Y9xY7H73_OsqCSY-IYEl-44-963000_ttl=0_create_MAJORITY_18884_21384_1643861149.88 2022-02-02 20:07:32,961 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_yEYtxpi_LY1FpC_XwX-axiOI2s3rm2mB7X32-vAdus9vKvKqjT8DhBjkZys5-qIvf_bsixcHIvXRm91yixejTuYjSz1C6mr55Gc87WAoJ1I%IidF0Us7Nl%D9xz3kcUHzh8omrhVj6tSuMKamgM6BuwPldsl_c5So-ng0pAgI%HMz5NEwrs_cen3qf5gtBNac0jK6S3hI-bGRYQReWKGrqvhLy11a44LIwGPP2p-44-946000_ttl=0_delete_MAJORITY_3256_6381_1643861149.88 2022-02-02 20:07:33,351 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_xo0khekOeFCwR6NU0nxWaVU_8IOk7OwP-IjGMyI_YM-g-jZM5oq6a9PeKnU1d%3Zgqtc7iywmAidXcfdc3NW-vKx4-2V8p2nI4z16SSDemfz%rWqwZA2B_0RNfYLYGbp-ZKl%IG9Lu0Mdrj9Qo-Oodsk-44-967000_ttl=0_delete_MAJORITY_3256_6381_1643861149.88 2022-02-02 20:07:34,203 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_HSrq5vwDj9-K6PFasqp99yMfi-44-970000_ttl=0_create_MAJORITY_18884_21384_1643861149.89 2022-02-02 20:07:34,204 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_K7b6H1E60992AHVqC28IxXJw3_5ju-44-947000_ttl=0_create_MAJORITY_18884_21384_1643861149.89 2022-02-02 20:07:34,523 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 75.23 2022-02-02 20:07:34,933 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_d4BaMPwHWvIwCDg%A9v4RbgevIMAQgCthFgwZsUfhgD5QFInCWBteeLdPbBVCzY4-ELqArL6p65_YJrxamjy-Y2ZpbUGYThZMlwk2bYTwlGXH-44-965000_ttl=0_delete_MAJORITY_3256_6381_1643861149.89 2022-02-02 20:07:34,937 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_1sGTQkZ0ttNfgPuadfXxmYgQXGDMo-pjs85OC7oGL-44-948000_ttl=0_delete_MAJORITY_3256_6381_1643861149.89 2022-02-02 20:07:34,938 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_-PNjTw-rHWn_92yoZUzXK1FMjGwMQJNqgSacDrKkzHB4Dtvp3i9SzAyIibo_Ezeoh2JIevTC7ugqdoagj3xUr_eC2fxj7PoMavYJ7SjJf5fntNBGsKjWDbai955ONyWNz0nF-44-967000_ttl=0_create_MAJORITY_18884_21384_1643861149.89 2022-02-02 20:07:34,938 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_q-NR1XAnBhIVy-44-949000_ttl=0_create_MAJORITY_18884_21384_1643861149.89 2022-02-02 20:07:35,966 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_gCey3hJBcRO7C0erb5w8IcZPb3D-e8eFU3IpndA4AmbKga1gyEpUzY2hUsTxNVm3frY5cxQ%V5ULBnkvhBsLeK3VM4oq74NmkYOJQ0yVnuuecRzb0cX1%G94%e3x_Zp-44-948000_ttl=0_delete_MAJORITY_3256_6381_1643861149.9 2022-02-02 20:07:35,966 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_Le4L6DSo0L4Kidap8HsClfoCUEZY1HIHfdASXHZwnYbaODxNrgXEfghXmehBn_MwwypWogZ8_F3iR-qC1kZIEjBOt5c8oFHTdHQthdnQXDqMb%9X1pNlRx1dWfX_akgb69Unpl9VQmuO-cadg-44-947000_ttl=0_delete_MAJORITY_3256_6381_1643861149.9 2022-02-02 20:07:35,971 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_vuLtPKrc_uO1I-44-944000_ttl=0_delete_MAJORITY_3256_6381_1643861149.9 2022-02-02 20:07:37,670 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_SdeXkdL2RgrH_sc4iM4sV9FBTYBTcIWwfwEeZg5NRfEo93XmefAswHj98jY01NeNvU22kxHdqezSLqYy_2qzRf7kLeyCd-44-944000_ttl=0_create_MAJORITY_18884_21384_1643861149.9 2022-02-02 20:07:38,460 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_UmN1CJgmM-44-959000_ttl=0_create_MAJORITY_18884_21384_1643861149.9 2022-02-02 20:07:38,460 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_xTgSYZjLmB9_Lhomk4f4l6KlrN7wmg74IPziULRlowTQfVRECrVQtHbo8OC7vP55nXL-6Fw_e9g8dicysvYneGgJbY2qYxHZt_5Rre64DglUPZI2vgvSla8WgVJxdzyDCKyQTom2HMJaknRnDNaqig5KY8SdVQbagez%30IQmzX-6dSMjZCXnvk1iaRmvgLAih-afvz_PnawlEuAbL9OFTEKvG9lIoM-44-944000_ttl=0_delete_MAJORITY_3256_6381_1643861149.9 2022-02-02 20:07:38,461 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_eL6jfEd%f6HD41i9_L%R0mvgcIzW8qQCP__g_fTpyppzJyAuVgAG_d3gDEpXR2xW4L3MiE0K-%mX0rOhi7favV30yWl9c-FgmXnqPwRw_ysQBCS_eRojIS3NjKiTgTEBGtKbhAVDHdBvRRd_-B8uUQFlJVCIxFYQDbs-44-942000_ttl=0_create_MAJORITY_18884_21384_1643861149.9 2022-02-02 20:07:38,918 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_ILdcaDzh6ddDdF5Nfo4eO7XTM-5SPx29UuvuiSel8q3fFnLFjjSBpllqH4lxdSwf9FdlCk8iCzGibudGccQT3eYE08QNQN20ri%VobxCw2ODE6gmyDUqGXfz%GuC9KBLafNrJlQJB0Vqz-cLqG82H3DzU03h14gxo-bqtk8d8a9JokI0-%xMSy9zOXL1RFTJTWZvLvahKId2ZlQqdc-44-969000_ttl=0_create_MAJORITY_18884_21384_1643861149.9 2022-02-02 20:07:39,282 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_2kfUhnhqbD8NNPYh6Gva7rqqjki%7m76ECHLQdu8K3lqX9Y5-aw8-44-949000_ttl=0_create_MAJORITY_18884_21384_1643861149.9 2022-02-02 20:07:40,631 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_RRKBLiFQuGTyACjtDuKVqegK2D%NbMZF8eh1sTLZSyDKFgeOb4GLN0aCtKp%vQ9HByk36ANeimElcjS51k52KGTq2p9XRi0tgsav8z9lP5eRSgOOf3qIb9m6JnOZ25Kkp3IYHEHOVCZ9rejp5awXpJfPREhIRGSNFbwX_GawP-QwLSKotVvsNYc3ZU-n2ABXnOwRVcATSxQ0opXR5tCrOcvjy0mj%nH5fyxDDsVM9USz3-44-942000_ttl=0_create_MAJORITY_18884_21384_1643861149.91 2022-02-02 20:07:40,631 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-44-955000_zInIMAId3F7uYTd3AeO-5Rj0f9Lc7MU_hDrYw7rzsInswgrbtTROdYtxpFIvKJi2mTtAadXtGIUZvmo4huPvt4%cdzz05vX%fFtkrdzh_IxEd7bTYYu_f9T5QVAhB8pyrfV-rLU4x625HpwYjOhl0qk0eL-44-958000_ttl=0_create_MAJORITY_18884_21384_1643861149.91 2022-02-02 20:07:40,631 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_4WL7PM9KlDIiqMOXiFS8n7A7icGIymJmNsjcbJxp4Ii0B1-04AjTLXoFx_fmXDWtqJRn1VCPBXBeFvnXlj0qfkSM_9YK-44-943000_ttl=0_create_MAJORITY_18884_21384_1643861149.91 2022-02-02 20:07:40,631 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_hAradTAvWpIOT8gT7_5qxZVpwIfXWMwGLJF4GP67XCqn-44-961000_ttl=0_delete_MAJORITY_3256_6381_1643861149.91 2022-02-02 20:07:40,632 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_IJodTcMctQxMvnZP2oSR0q-p1O7lucMJJQNcDDp_NgqSOuaC8axSBC6-44-942000_ttl=0_create_MAJORITY_18884_21384_1643861149.91 2022-02-02 20:07:41,069 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_jDmmG7NKAbHBaCBEYXUjQhnhmntw2S9n7IS93gndPyYxxmh9oRhAxsYXUxPWPiPfncuU8fT-BfwmQpWeCAlbM__IVvj%ehHuEa-AG_dN0t0-qvloAooktSuO0yKsfkPUgWXMJW2M880wjf_Wf2n%PLM0bk93J4itgoEn0Z_wzdepiDRys6GDQ3OZwwX-44-943000_ttl=0_delete_MAJORITY_3256_6381_1643861149.91 2022-02-02 20:07:41,069 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_p8tx1gRKsyJWa1c%pU7pN0pnB9Yb3mqrqYABQpRii1PEKnSkdTuK3cCq9fO5Runhzbpkurc4BW4awcb1N4y0RBl%C28_eMR02PM%aV0ouRwGeUd5h7MWeZjcl0EynAX9dKNe8wKn_cmEQ2FOG8r56ILR5p9mggZrUo209pJ5FmLITl40IfXYz7Lr%nAzB2-44-945000_ttl=0_create_MAJORITY_18884_21384_1643861149.91 2022-02-02 20:07:41,069 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_NklaorkV-9Ze0Cg4hFedp1Q3t0CZOeJ1zcLOHk6MOnphHCYHr_-uZkFxxfp_Buiyaj%PJT%p7k8nRwLx8VyPVhLamfIgNN%eVfsjxh-F6i6no-3M8CYUvVr%sfxMyWsOb%%afG5r1IaI6Nzz1PWfC1XBow8BCv%SttNayNMyzUavjTR01UDy%5as8hNLe3oR05Crm9gDpf0cCpASZdJ84r1xA-44-944000_ttl=0_create_MAJORITY_18884_21384_1643861149.91 2022-02-02 20:07:44,565 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 77.13 2022-02-02 20:07:44,798 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_QYnP8VF8SpL-QFSgGcjfpg_nSFiHH8utmJtpFrgiduqjAnigUD%3Lz0xLLnNpcAdiwmtj3FMKSST8WrO6zxZAnDsbh-QicTH-oiggUZ4YH6xvDw_kAwpW8qrqbJTFjOBqQgTY3L0ip-W0KAiG11PniYGSB4iuvc%4yNOf67WayImBeavEYB-44-960000_ttl=0_create_MAJORITY_18884_21384_1643861149.92 2022-02-02 20:07:44,799 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_PZ%JsKPT%c%Bl2Hb0FXcRrgpu8SvWrzOx-eQ90JhHxLBLbbdsxHMBQE32Nto84ykUEbScO_5AdHAnaAp5KR7djIZrqP18xYizYAao5VJ8bS1nvY3okhL1QFBczKnFp96cKCikIjgDQT-44-945000_ttl=0_create_MAJORITY_18884_21384_1643861149.92 2022-02-02 20:07:44,799 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_cX92VRYp8GUs-b_thEGRwEFRsLzugdQXbV2A7xLo0DpVBg2qpVZYOeoQzU_KdOHI%gZKMczHUzooCLZbsj8wRsyWziGfrJjUr_Zm-44-964000_ttl=0_create_MAJORITY_18884_21384_1643861149.92 2022-02-02 20:07:44,799 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_yEYtxpi_LY1FpC_XwX-axiOI2s3rm2mB7X32-vAdus9vKvKqjT8DhBjkZys5-qIvf_bsixcHIvXRm91yixejTuYjSz1C6mr55Gc87WAoJ1I%IidF0Us7Nl%D9xz3kcUHzh8omrhVj6tSuMKamgM6BuwPldsl_c5So-ng0pAgI%HMz5NEwrs_cen3qf5gtBNac0jK6S3hI-bGRYQReWKGrqvhLy11a44LIwGPP2p-44-946000_ttl=0_create_MAJORITY_18884_21384_1643861149.92 2022-02-02 20:07:45,585 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_H2lz47gg2FfNjIGKNEUFuLEefWp4ntHROEFKf_gCLAKP7TMVd9YJ7pfSp_cPzZ2M6RfIL6xRRYnh94EsXAJppjV2DBWLi7xMEC6btGnxcB0FALF1xXnwfoHUQfwFfpEx%Ha_fq%lZUiR7WamHotAfZGP329C7mHmPfaqUyaCp-44-965000_ttl=0_delete_MAJORITY_3256_6381_1643861149.92 2022-02-02 20:07:45,585 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_38K%RGGaUb2bnFf18iJePHECs7bt0ZamfBiOp5XWDjQKcl6P2Z7axGK8AQRz2QJPYM1dmy8Nr5iCB%Q_V3ZKZCCxA0b8h-QKHGN0pAhrNFnC3u-44-969000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:45,585 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_q24Id5CGuJ4Xjv4X4Yu_EQ6uQwkL2DGCu%lCRePboapUqBD7q_L_nIv7z6eGj-Q9GQM3Zt3tHtyos3n19BMbv%CAHeI9rnczcxPvJe5FsEJJ-MUBf8yOSDrNf64WO1wd9Z31IZ-3U%BIfrST1mWSXYik2rryejdbjBXVOKhD1IO5RiXuRXqUT45FaSCvBeoc0PbvponaueyI-aHQ-44-971000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:45,585 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_oldx-44-960000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:45,585 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_RX57TAtOUVaVpEMtg_kVnjWqAP68pwG94tkDDG_18WWMj14OxBxsDHDzjLSpuZUM8p4dKDQLBPV01yHoKXG_h6GSdfWVMqoxDY_bwrbN-44-961000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:45,805 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_aOi346RI7BGSZ4Y7n%HSw_XPB-66XVUD_4GypZ9AGRcLvJ44EIAL4VJtMkEXuPs3IaZ7i_rNrFs5EvgJJrj-8f8oLGCiXbCCfBfhv1Zea1lOTU-44-950000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:45,805 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_49qOJfOkdDkouWflxH8ntPUe0VbLV-otnljbaIpGFXL7wNB8wrbBjzxj8%CRlnxjp-1u%rVnsX60%v-tjzmsHhwCMRXYyLaO6_Vy9aNm7GFvY-KTRC6npMbO36k3U4dMngdJtx%0MRQMdFibdw_wsluhAGTxBE2jk1EoAC%EPReiOd-uNOEw9Ov2eeXju_GHQro02JSN%as3LklX6Q7yr2xpAJ0CZmykZW-44-954000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:45,805 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_WgdwfBvuBjftxAsspW9JwYJ%1t%9f9zGEdC-44-968000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:46,565 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_4-pMjiGeoxZqLxRg9fDbWc1Kx1dSfv8DP7DLD8bii1ridz0da2oqUWOfZFdr_VqgfmnXIsDuVWVqJBrMr--5AjvZhQxs_N8PofZz1oH59M98OIRVxBC5Qu-TtQp2t3yC6RpCaGIppVWl6dWfQamhiFASO-BGLbFZnrsjT-44-945000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:47,678 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_7Th%ue6REVzE251rI3cK9nomuXb0ZRzlsjCPlrD5hXsB9gmHZ2PHHzfeDUqFfg1hF3YnEqPx_C9ZCGoPQBcIhF541GUdTtqESGFZT037-44-948000_ttl=0_delete_MAJORITY_3256_6381_1643861149.93 2022-02-02 20:07:47,680 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_pZeILWeNMUe998_W0nSdMZpl11CAhkk8IjbWpharcEXQ1uKrlN5A7yr-_hArrEjJlaVInAOuJZ0h9kmDlykrCIEuafa7Cy8wViQOOLGUn4KhMS3LI95gla7aarDHUFeS8bvMVRa%6GJByGnj6ELX36KUM_XEgyCeuqRL-%Jg_e7_Lz%wRlwxfdP21Au37HW0fqyhKHfYlU9uBvG13uHo9Lw9oeG9vAl6ld1-44-944000_ttl=0_create_MAJORITY_18884_21384_1643861149.93 2022-02-02 20:07:47,680 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_tfej2VFzngmD-Q-c0PlP9fxsh25bNhZz2%9NzVHUChmzcrTAtK6XbzmDxQfez%RCDjEj8tQLST2%ACY9dPegq8Y6ydWyM1IruIGf71bE8xrVHpX-c%g2NZCTB6uGVEQ8ILiXWo9DyN98X6IHOPbBM3L1kPnQQnT-Lt5oAHk5ygDzVz-6a81B3D3cRCYE0EHBZMCgsNkDL9ZBPA-%8VqN-44-955000_VCrhYnejPzZIMSRA_Yr-ZPcZ0p9MrmPRIWksAyFi%_7iAkE7wgYusGLBdgvVpR3Nkq2N1-44-958000_ttl=0_create_MAJORITY_18884_21384_1643861149.94 2022-02-02 20:07:49,036 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_F-2EI2Rh-xTnKcheDnqBdjS7093On1_ycXNhthhUxfvwL2httR4_w0XhTeDTg%TyHBt71IFa147tegA8O5GaEbQqjFPozl405hmduVGOFDXnd7lBuRIVNReij6m4W4o0DViZmnaXzcu4llq06ZBL1Q8KjinjD%eEcq-05SRehYeipfFGVDiP-Efs60i7-OL4-44-949000_ttl=0_create_MAJORITY_18884_21384_1643861149.94 2022-02-02 20:07:49,036 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_d-4d0W78bqxC8THAnvyAaVtz58X7iBjuZ3eRq1AHXeNH1yp7rfNpCh-44-950000_ttl=0_delete_MAJORITY_3256_6381_1643861149.94 2022-02-02 20:07:49,457 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_g0YmCgrxY5H2Dm0qeHAvUqsFa_My4K2PA6e7Cm%gyJr9p8Eukbg02XpJPxblIVZ7TkL6wqC4vRf36NTmnkDfn7JsuNQeUCitHb30e2NtvmIhOp2UvxixMZiN2xXChkCF_iqTfERY5JQtji3nvoT-v1BPuz9DuX%lNwYNiOdB35oJa_7E6TGcsgAZXXHQkPsi-44-941000_ttl=0_delete_MAJORITY_3256_6381_1643861149.94 2022-02-02 20:07:49,457 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_CDROhTBLNzWMXLUdbAJX1VQJjPJ8AuE6zPqPSIEO0SpeAl83y7uu1qMJ64mhxri9uRkJWvpBkYUJLyQcsfxwDucTaPZ5gFyIDPmdxpQ2ZjJmTgZcCPbvTdgdMT13V6SyMgj0WNVQbomb_oDDIQrJKTjORlsyn1UGE2LkIZJTyvHQ-44-943000_ttl=0_delete_MAJORITY_3256_6381_1643861149.94 2022-02-02 20:07:49,980 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_5VIw8HTNqY7PlEDVlBnQIMYCjyUwmXK8lOz7_tQ7ZWgWwbEtIByf%4XwB0dvgwUcX9W%gQcnPtwIodTFncjc%%yaYYU%-44-965000_ttl=0_create_MAJORITY_18884_21384_1643861149.94 2022-02-02 20:07:49,980 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_vau-CndB8BNbKEOrErJcS4fgGL_X_CIafo85WH7EH-OfNVqX2zuxbqJYQD-uA68MYvLQnT-1ZjzzhbxCO-44-966000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:49,982 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_Rxbn6qv%LVL_WPM%yqX5lWLX0UUJmjqO15IqEi7QaF1hWihtQwMMzkcy9cjRz18EIL6txwbos_sbv%2-44-951000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:50,447 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_hX-44-950000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:50,447 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_FeP8tGX70T5LzXtvuliLLnJeH88Kbwvau2LvSUQFyee2pVZZndLSEA_0_vemafEflvx48aVdyd82rkMqCIgBucLVDvGcVkwh%J-aFsv6vaoej_u6iNYLuVCclqCY3c68IybIGXx3uA3G73Z9QIZIlUw_JDI-44-970000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:50,448 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_gVdJH5wLEKdfdYJNvbfRsnpN07kkGna7u3z60t_YR01iArtJ6JPXAkmXpqu28DzbY17XFepTj9EKn22TLDuulGU2rEXS5qhTHhKp0Q-44-947000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:50,450 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 3256 to 6381 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_jDAH-IQwLyPunSPd3XyRJac1fGszRM9FeGsRgaUEQGQyAxBZfr9VY5swiM1IAv1ZnZX4BmsCJ9jCx4S2M5LYBfx7TD454y8h7fveCqmHYkPoPyor6eQUmePLh5JzeVabDUkY9kPr4U3n1ZDh-8_GtqBH_ImIZ_%nnjRLFFy72APOa-Tm4UPZ-6sJKijBeTFzyUT56FpYWK85BI3nl0pZru0ZWaczgzMO3-44-971000_ttl=0_delete_MAJORITY_3256_6381_1643861149.95 2022-02-02 20:07:50,746 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_SkxPpc70ZR6I4Jm60Y5M8OMIatWliXfVGlZNruJOizO6m92q_CDSFWbzZl0FEo_OEIyRCJykSCm3a0ohOjXw1icyZJEkw%MjQmlLAa00ezNkHd-0KmmarC6I%FCzyM-44-964000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:50,746 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_kcXWiIKiodtIk9LrNL1%IFsZNL2HvQBZg4l0ZIFZS7VqY77Q2ghcPye5WF9tWG-AiDmpNHvscHVoBkY9Rk7npE%sR0ddBU1gr_9H31Omzl2bi-s6PMIJRoZ3K-MeJgMG2TKOEaDrfR-RdKlVRZONL_%bGiVyNVRzWM4MBPWTepvV5lgeMB9fzXcj-ov57vPXm-DsuuWTM%qMJ-qb3qQXkOnpVJh5Fj-_zPZzxd4sgvRjjYlO-44-959000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:53,002 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_rGDE42zlKooE-7SKwaZik1739jqzLYaCOU99CS%RVL51hKrcdjFgXamSjN3CwZ0cjc-44-968000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:53,003 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 5 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_5Qb1555D-QNUCfFoXQUufqSio8FmUL%RQjh9F%aVaRV8092xdN4Zd%zds9hWJ-R32q5Odx_glvpNIl0H7nvZ5-X5urWIlxjJol-44-944000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:53,003 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_UNoXXKkBAwYB2tdYRjiOLetynMY-_NAvSPJK3-8CRE3Acvpi6n6TxTEWazOIJzf9Ana6lGbJAWD-qlAEs25JmFgdmpJabI0nbnCWx-ymM2wtx-44-949000_ttl=0_create_MAJORITY_18884_21384_1643861149.95 2022-02-02 20:07:53,003 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_rpIkqLN6r75DQxSSExPya0qb7URAWsHzT1XlrL7UwlHsogefomElzr8TOeZiyD_TnSjtLlFGxPnyqLAQFj%q4npNfEf4T1SSQaBhNGRkoAkyuDwCprNd2AyFwbSxLe0n%IigyFao6fkkUhTDlg6-PZjwjLqB9kMKycvtoj%VUYEftwCr6xGtw00Ot17y1uEuYgYyYcnN5y6MEIY3KSwi7gwLuWJXuitfH%-44-948000_ttl=0_create_MAJORITY_18884_21384_1643861149.96 2022-02-02 20:07:53,723 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default__default_hssGjor9A2Qz2-44-943000_ttl=0_create_MAJORITY_18884_21384_1643861149.96 2022-02-02 20:07:54,634 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 78.97 2022-02-02 20:07:56,575 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_jxczA-ukqJb9QUSqqk4jtfZAZGoJcLAojtxsDvZ19faY_7y-44-944000_ttl=0_create_MAJORITY_18884_21384_1643861149.96 2022-02-02 20:07:56,575 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_vsXqDMekmQ%W4MH4FpLYbdGSWTmd41sH0nmVH7Bj7b_6km8vJcZE-idGnoz2XAergyfHSstsnJEK8ZamCs1zOB8VUyYHq6W%kY4ge4KZ2_gBJ1lER4IVmYauzbL-viP77XMKB7A3mHXSuDUGDEIBm-S7YtdLB1Jm6KwBwk0R_dlyXFjRRvXeYsBgZ6dDeMARtg6R6qlYwCKasybt6cthvbH0-44-965000_ttl=0_create_MAJORITY_18884_21384_1643861149.96 2022-02-02 20:07:57,289 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_hAradTAvWpIOT8gT7_5qxZVpwIfXWMwGLJF4GP67XCqn-44-961000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:07:58,678 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_vuLtPKrc_uO1I-44-944000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:07:59,568 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_L%YwH-41%6wglO3S2SbfjBf3j8vcErAQQCiSSR1JMe73R3VEZUo-kvVBlr_-Heo_p4HWb0mUYiYXS%Gdt6StsxdxKR5upD3VuewpBd38mYw2gWhdxSjiJfOctE6raVRdUSgmm%cO3%D_xm7NqzMwHmEV-44-943000_c2Q3M0_VZE2vY6ytsJ5R94H64F%PjnctX96XyJLoqe21UXWtoZ5gR%n%dKIXdUwFccfdz6J62TXu%1dmID91Ok8UzDbd9NTvw_GV5ZFFX2F-GF5Pi--44-946000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:07:59,568 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_gCey3hJBcRO7C0erb5w8IcZPb3D-e8eFU3IpndA4AmbKga1gyEpUzY2hUsTxNVm3frY5cxQ%V5ULBnkvhBsLeK3VM4oq74NmkYOJQ0yVnuuecRzb0cX1%G94%e3x_Zp-44-948000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:07:59,568 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_duiKfQ1or8O6wAbISuBNa01PI0bA4z2a86DIc-44-950000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:00,096 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_jDn2va74fa2OFl2q3DuMG4OYxfGw3tsWeYgI_Rn-g2_Xb7dvsLq61WEumDNPyPiB9meYNN_cs-GLv8teu8P7J3FLYfmGAlcu04E-MtDRN5A8j8uCybp7BOqVyQ9wC%2FwzjJk8HYfCGyXRqTwLpzk%xGoIYoUgWwOcy2IAG2N8wTEFPoFcpnSCAVzR8MPyyUhr5FKTuC9dBV%mILHeB76Hq_71rmEyid2Do_wt6fdpM-44-952000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:00,098 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_sRvhoOsuQwbG1iFhEJxUygcMigPUEaLF-sLzY0hMQCWyAP838fI7jCEQi1wgfrJubGJeUH%4OGpg7KuZNm3ISqAilUsdNEdBBY-XxJWkVlmjNAzLTfoIiLPAIvOiuC5LgCyBcLAcYZYIIX_lgEC_io-RVMzUNeiOg9XZPgNodSeyZubD%aeEP3oKOngQwwMixJ6lQGUrJ7T-44-964000_3IejvdWpvm6t2Slh%Cntkx_xpLKWzLALKNOZWAK7IUQV7cBYzYBKtgYo99dC9RefLwnysChnVkAxauyvx_O6Pxcu5amLX%ua4S9m6PR6Gz_NAWWhuVmkcjqV-EtJ6Lvtj7ewljSatSmSHr48OLSRigMy9M8GmgAnwieMVwVIjMEerYXumrrm_x00Wt-44-968000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:00,973 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_C6r_5umgeQrB2UblNlN4jIl8BQ5hmoi%WEwL_BtfGihk%uRhn4T1ocWT2YV7r67ZtoRPtHpVaLcr13hE4ZAu84vnrrgw_rNEm1zqtNDBKd0HgSb9ttOic6xVLXL8qnfJRVW_AOErbc0rgQIzE%qJy2Fm_sgbcHPT_kG42t2%Gzh_3dVXMT7UUsNhyvoqv-44-970000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:00,973 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_bcnfAaPtCInRECBppHpqBtcCDw6JSO2scnSCz_gek5nEzGphzokKD0ogCKXlYmWM%CvplPbR_ikRNJCf8fYDTYDwRmAmMZB7aAjbu2k-Nn-juJN6p_fm8pY_OlRGj%Vu8-xZN_OgPZh92mX-qyZTn9q%5yqQf4W4hVy4iXy9xqv0Uh8SObp-44-953000_rjjnoMhS9eUGdI-44-954000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:00,974 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_U8ZeqYZnqWA3P4YNPmnWUN5kBaFz0_1Al2KtwRvhJqSsa9zhCj7z-44-962000_0PkpIeYZa%Ln_yGjGE0UM39cX1RP4ixk5hqY8br-fUtwAlI5XF1m3g1zwerg7xdB5%oxA0OBTJmF_dKEicmK0T%w2Js7d3yVlHZ0-44-964000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:01,342 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 2 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_dLrWOIIHZSz_EuINKGB6mrWaWwTnqluyfr2WnoA2AVZvvRy4YTnXSLK4iaIxTkVCMrY13KXMiTdBRj_Q-DdQCwAQxBh-v--44-959000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:01,342 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_gYtdRKkEHiGb0N0b-44-958000_ao0MYmcFJF7HuMAE5Pkt6n3FG_ik4rMpruMVOEYRi2no9pMblSesHVv1KcNFXnCyPdfR-KVy_N0Ff5bawfTbHzdK5t-hHqDvXObco_-HrRExSADomrO-44-961000_ttl=0_create_MAJORITY_18884_21384_1643861149.97 2022-02-02 20:08:01,344 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_Pbwv%oUOgkDFjmp2gH6oAM9UkPqrV0mn8GsXqQcbKk0HresdaeSGHVCckqBI8ziK5G-Y52QlDx8tSM5%aA%da29i-wK9rAekMTeVAEU5ZnXfrzWuKoBcXjvedPIELV%_muEO1%M5X0PKdDOf5c-44-971000_ttl=0_create_MAJORITY_18884_21384_1643861149.98 2022-02-02 20:08:01,344 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_njSdk96IycYStdr0dCe1J6nwhBlOqqn8dtG22Yv682s%%ERg33fHaoMDDU%eSwyIb6t3une5c7YxgNCdAiyranUISjfgWlul2bQ-zkJPQrWP%9tfQNwB_Eo-44-950000_ttl=0_create_MAJORITY_18884_21384_1643861149.98 2022-02-02 20:08:01,605 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_thb_4JiGBBIBs%XunLR83_AovgZpLjRpXI3WzLInY8Cj--wm332X9rW1EvdNMOARwiEyg9LUWKSAFO4XYpTXPN8kY4yWTOQdoagDtV-44-952000_ttl=0_create_MAJORITY_18884_21384_1643861149.98 2022-02-02 20:08:01,607 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_Ms1c40UZEXJ65dKIm3BgezoBmgi6-zy5A6z3CMYiP0pH%-44-950000_G%-ZCtQGNz30rwOy9-1GXqMUWMNuh6jjE3QMsFvkhji3i6y6dTr1C4bhwRnNQdS98Q-44-952000_ttl=0_create_MAJORITY_18884_21384_1643861149.98 2022-02-02 20:08:01,607 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 3 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_UzYk8ACfLs1JrG9oG3fL5xNbHZE%O1Ro8vnt4dW6Sx70gzMXIAeVRpjM0HEbaM-53WgCwAQIvQJsugYDesOzrWKlzktdnf2fPHYcB8GNH1p5-44-968000_jDAH-IQwLyPunSPd3XyRJac1fGszRM9FeGsRgaUEQGQyAxBZfr9VY5swiM1IAv1ZnZX4BmsCJ9jCx4S2M5LYBfx7TD454y8h7fveCqmHYkPoPyor6eQUmePLh5JzeVabDUkY9kPr4U3n1ZDh-8_GtqBH_ImIZ_%nnjRLFFy72APOa-Tm4UPZ-6sJKijBeTFzyUT56FpYWK85BI3nl0pZru0ZWaczgzMO3-44-971000_ttl=0_create_MAJORITY_18884_21384_1643861149.98 2022-02-02 20:08:01,688 | test | ERROR | pool-16-thread-29 | [task:execute_tasks:3940] Failed to load 1 docs from 18884 to 21384 of thread_name LoadDocs_MutateDocsFromSpecTask_1643861149.03_default_dpKlbKKQQ7Qb%doj0HpvdEP2qGaIiMi6yro7at5H7lHV2p1kvtbIzakJRGymDtdu5GoPoxdshbbaNoUEChI9NUnm9DjqxA0un86eHU_dSefp0g2ar-wItV1Gu3ifHxvq0pXbxs%aqWmeMrVH7RM5FoYoaZrnF8xEwCwdA1x3tzASw_8EcB%aiMcW4tHKxV-44-946000_ZFe18IkMKsgH9QZOPZC1W1PKeICSizfISvkFzJl34PRTs_xnj3g%atLME9c4MjvTwfEsEhHJXwJCYyUK7G6NpRDF-CmojH1LJ%gh%IdszZgWXSkoR7K2j95bc-a0XatLQp34wM2RNYFQWT4-gZ-SV%S%oZHxxZ4lFRvZe0s1gN-44-947000_ttl=0_create_MAJORITY_18884_21384_1643861149.98 2022-02-02 20:08:02,717 | test | INFO | pool-16-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.............................**................******.........................................**........***X | 33303.900 | | 33303.900 | .............................................................................................**......*X | 35110.900 | | 34465.000 | .................................................................................................*****X | 34465.000 | | 27176.700 | ............................................................................*X | 27176.700 | | 25492.500 | .......................................................................*****X | 25492.500 | | 23691.100 | ..................................................................*X | 23691.100 | | 23248.600 | .................................................................*****X | 23248.600 | | 22434.100 | ..............................................................*X | 22434.100 | | 21525.200 | ............................................................*****X | 21525.200 | | 16962.100 | ...............................................*X | 16962.100 | | 16189.500 | .............................................*******X | 16457.000 | | 16350.800 | .............................................*****X | 16350.800 | | 14854.100 | .........................................**X | 14854.100 | | 14380.500 | .......................................****X | 14380.500 | | 14100.700 | .......................................**X | 14100.700 | | 14042.200 | ......................................*****X | 14042.200 | | 13283.600 | ....................................*X | 13283.600 | | 13096.500 | ....................................**************.....******X | 15002.100 | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 20:08:03,757 | test | INFO | pool-16-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***........................**..............******......................................**.............******........**...*****X | 463.100 | | 384.100 | .................................................................................**X | 384.100 | | 323.500 | ....................................................................****X | 323.500 | | 265.600 | ........................................................*********.......*X | 301.000 | | 290.600 | .............................................................*****X | 290.600 | | 193.400 | ........................................*X | 193.400 | | 178.000 | .....................................*****X | 178.000 | | 149.400 | ...............................*X | 149.400 | | 127.500 | ..........................*****X | 127.500 | | 95.900 | ...................**...******................................***..****X | 264.900 | | 204.500 | ...........................................X | 204.500 | | 181.500 | ......................................**.*****X | 189.800 | | 189.300 | .......................................*X | 189.300 | | 170.100 | ...................................******........**..*****X | 215.100 | +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 20:08:04,296 | test | INFO | pool-16-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***...................**..........******....................................**...............******...................*******X | 838.300 | | 793.800 | .............................................................................................*X | 793.800 | | 729.600 | ......................................................................................*****X | 729.600 | | 523.300 | .............................................................*X | 523.300 | | 508.100 | ...........................................................******...***..****X | 551.900 | | 486.800 | .........................................................**X | 486.800 | | 409.400 | ...............................................*****X | 409.400 | | 387.300 | .............................................**......******....*X | 477.100 | | 455.000 | .....................................................*****X | 455.000 | | 435.400 | ..................................................*X | 435.400 | | 402.200 | ..............................................*****X | 402.200 | | 213.200 | ........................*X | 213.200 | | 197.900 | ......................******....................***........*****.....*.*X | 484.900 | | 436.800 | ...................................................****X | 436.800 | | 239.400 | ...........................X | 239.400 | +---------+--------------------------------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 20:08:04,688 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 80.33 2022-02-02 20:08:05,566 | test | INFO | pool-16-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------+-----------+ | 11296.800 | ...................................................................................................X | 11296.800 | +-----------+------------------------------------------------------------------------------------------------------+-----------+ 2022-02-02 20:08:05,576 | test | INFO | pool-16-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------+--------+ | 96.400 | ...................................................................................................X | 96.400 | +--------+------------------------------------------------------------------------------------------------------+--------+ 2022-02-02 20:08:05,595 | test | INFO | pool-16-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------+---------+ | 239.400 | ...................................................................................................X | 239.400 | +---------+------------------------------------------------------------------------------------------------------+---------+ 2022-02-02 20:08:14,723 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 83.36 2022-02-02 20:08:24,759 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 91.65 2022-02-02 20:08:34,795 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: running, progress: 99.46 2022-02-02 20:08:49,838 | test | INFO | pool-16-thread-19 | [task:check:409] Rebalance - status: none, progress: 100 2022-02-02 20:08:49,881 | test | INFO | pool-16-thread-19 | [task:check:468] Rebalance completed with progress: 100% in 185.858999968 sec 2022-02-02 20:08:50,213 | test | INFO | MainThread | [collections_rebalance:get_zone_info:968] nodes in zone inside get_zone_info():{'Group 1': ['172.23.105.164', '172.23.105.36', '172.23.105.33', '172.23.105.206']} 2022-02-02 20:08:55,470 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4707] Validating collection stats and item counts 2022-02-02 20:09:07,509 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 20:09:07,510 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 20:09:16,219 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 20:09:16,219 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 20:09:25,006 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4570] Bucket UID mismatch. Expected: 0, Actual: 1 2022-02-02 20:09:25,006 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4724] Bucket manifest UID mismatch! 2022-02-02 20:09:39,901 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ | bucket1 | couchbase | couchstore | 2 | none | 0 | 92100 | 7.81 GiB | 262.77 MiB | 374.06 MiB | 100 | | bucket2 | couchbase | magma | 2 | none | 0 | 46000 | 3.91 GiB | 513.24 MiB | 579.03 MiB | 100 | | default | couchbase | magma | 2 | none | 0 | 3750750 | 2.00 GiB | 1.49 GiB | 6.87 GiB | 57.0277677798 | +---------+-----------+-----------------+----------+------------+-----+---------+-----------+------------+------------+---------------+ 2022-02-02 20:09:40,384 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket2, Active Resident ratio(DGM): 100% 2022-02-02 20:09:40,384 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2022-02-02 20:09:47,263 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: bucket1, Active Resident ratio(DGM): 100% 2022-02-02 20:09:47,263 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2022-02-02 20:09:53,869 | test | INFO | MainThread | [collections_base:tearDown:82] Bucket: default, Active Resident ratio(DGM): 54% 2022-02-02 20:09:53,869 | test | INFO | MainThread | [collections_base:tearDown:85] Bucket: default, Replica Resident ratio(DGM): 33% 2022-02-02 20:10:49,865 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 20:10:50,285 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 20:10:50,286 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 20:11:20,428 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 20:11:20,854 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 20:11:20,854 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2022-02-02 20:11:50,970 | infra | ERROR | MainThread | [Rest_Connection:_http_request:286] DELETE http://172.23.105.164:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2022-02-02 20:11:51,369 | infra | ERROR | MainThread | [remote_util:log_command_output:3013] wc: /data: Is a directory 2022-02-02 20:11:51,369 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1612] Unable to get timings for bucket: global name 'StatsCommon' is not defined Build was aborted Aborted by Balakumaran Gopal Archiving artifacts Recording test results [description-setter] Description set: Finished: ABORTED